Skip to main content
Glama
Sign In

Spicrawl

by com.spicrawl.mcp
OverviewDiscussionsAdmin

Submit a batch scrape job

spicrawl_batch_submit

Submit many URLs to be scraped as one asynchronous job, with shared settings applied to every URL. Give EITHER urls or items, not both (at least one URL, unless open is true). Returns the job object (id, status, estimated_credits, progress). Poll with spicrawl_batch_status, read with spicrawl_batch_results or spicrawl_batch_task_content. Use this instead of many spicrawl_scrape calls when you have tens or thousands of URLs. Set open: true to keep the job accepting more URLs via spicrawl_batch_add_items; an open job never finishes until you call spicrawl_batch_close.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "spicrawl_batch_submit"
  }
}

Input Schema

NameRequiredDescriptionDefault
nameNoA human label for the job.
openNoKeep the job accepting items after submission (append with spicrawl_batch_add_items, finish with spicrawl_batch_close).
urlsNoShorthand: URLs scraped with the shared settings and no per-item overrides. Mutually exclusive with `items`.
waitNoMilliseconds to wait after load (rendered scrapes).
itemsNoLong form: one /v1/scrape request object per item. Besides `url` and `external_id`, any /v1/scrape field (e.g. `js_render`, `wait_for`, `response_format`) overrides the shared setting for that item only. Mutually exclusive with `urls`.
formatNoOutput format for every item (`response_format`). spicrawl_batch_submit defaults to 'markdown', the same as spicrawl_scrape; an item's own `response_format` overrides it. 'json' returns the raw HTML document, not the markdown `content` and `sections` of spicrawl_scrape.
renderNoRender every URL with a browser (`js_render`).
max_costNoCeiling in credits for ONE item (not the whole job — see credit_budget).
priorityNoScheduling priority of the job relative to your other jobs.
wait_forNoCSS selector to wait for before capturing (rendered scrapes).
concurrencyNoMax items in flight at once for this job (may be capped by the server; see warnings).
max_attemptsNoPer-item retry attempts on failure.
credit_budgetNoCeiling on the WHOLE job, in credits; the run aborts when it would exceed this.
premium_proxyNoComing soon: not available yet, do not send. Use residential managed-pool exits for every URL.
proxy_countryNoComing soon: not available yet, do not send. ISO-3166 alpha-2 exit country; requires premium_proxy.
custom_headersNoExtra request headers sent with every URL.
block_resourcesNoResource types to block while rendering, e.g. ["image","font","media"].
failure_thresholdNoAbort the job once this many items have failed.
main_content_onlyNoStrip each page to its main article.
webhook_endpoint_idNoId of a registered webhook endpoint notified (with the job object) when the job completes.

Output Schema

NameRequiredDescriptionDefault
idYesJob id. Pass it as `job_id` to the other spicrawl_batch_* tools.
nameNoThe label given at submission, if any.
openNoTrue while the job accepts appended items (it will not complete until closed).
paramsNoThe job-level scrape settings as stored.
statusNoLifecycle: queued, running, paused, cancelling, completed, failed or cancelled. Keep polling while queued/running/paused/cancelling; stop at completed, failed or cancelled.
priorityNoEffective scheduling priority.
progressNoLive item counters and the credits charged so far.
warningsNoDecisions made on the caller's behalf at submission (a clamped concurrency, items not yet queued).
error_codeNoMachine code of a job-level failure.
project_idNoProject the API key belongs to.
started_atNoISO-8601 time the first item started.
status_urlNoAPI path to poll for this job: /v1/batch/{id}.
concurrencyNoEffective per-job concurrency after the server's clamping.
finished_atNoISO-8601 time the job reached a terminal status.
results_urlNoAPI path of the job's results: /v1/batch/{id}/results.
total_itemsNoNumber of items in the job.
max_attemptsNoEffective attempts per item.
submitted_atNoISO-8601 time the job was accepted.
error_messageNoHuman-readable job-level failure reason.
estimated_creditsNoCredits reserved at submission (a hold, not a charge; compare with progress.credits_charged).
failure_thresholdNoFailed-item count that aborts the job, when set.
results_expire_atNoISO-8601 deadline after which results answer 410 Gone.
cancel_requested_atNoISO-8601 time a cancel was accepted.
estimated_credits_microNoSame as estimated_credits in micro-credits.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • changedInput schema / properties / format / description
      Previous value: -"Output format for every item (`response_format`). spicrawl_batch_submit defaults to 'markdown', the same as spicrawl_scrape; an item's own `response_format` overrides it."New value: +"Output format for every item (`response_format`). spicrawl_batch_submit defaults to 'markdown', the same as spicrawl_scrape; an item's own `response_format` overrides it. 'json' returns the raw HTML document, not the markdown `content` and `sections` of spicrawl_scrape."
  2. Changed1 schema field changed
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "additionalProperties": true,
+  "properties": {
+    "cancel_requested_at": {
+      "description": "ISO-8601 time a cancel was accepted.",
+      "type": [
+        "string",
+        "null"
+      ]
+    },
+    "concurrency": {
+      "description": "Effective per-job concurrency after the server's clamping.",
+      "type": "number"
+    },
+    "error_code": {
+      "description": "Machine code of a job-level failure.",
+      "type": [
+        "string",
+        "null"
+      ]
+    },
+    "error_message": {
+      "description": "Human-readable job-level failure reason.",
+      "type": [
+        "string",
+        "null"
+      ]
+    },
+    "estimated_credits": {
+      "description": "Credits reserved at submission (a hold, not a charge; compare with progress.credits_charged).",
+      "type": "number"
+    },
+    "estimated_credits_micro": {
+      "description": "Same as estimated_credits in micro-credits.",
+      "type": "number"
+    },
+    "failure_threshold": {
+      "description": "Failed-item count that aborts the job, when set.",
+      "type": [
+        "number",
+        "null"
+      ]
+    },
+    "finished_at": {
+      "description": "ISO-8601 time the job reached a terminal status.",
+      "type": [
+        "string",
+        "null"
+      ]
+    },
+    "id": {
+      "description": "Job id. Pass it as `job_id` to the other spicrawl_batch_* tools.",
+      "type": "string"
+    },
+    "max_attempts": {
+      "description": "Effective attempts per item.",
+      "type": "number"
+    },
+    "name": {
+      "description": "The label given at submission, if any.",
+      "type": [
+        "string",
+        "null"
+      ]
+    },
+    "open": {
+      "description": "True while the job accepts appended items (it will not complete until closed).",
+      "type": [
+        "boolean",
+        "null"
+      ]
+    },
+    "params": {
+      "anyOf": [
+        {
+          "additionalProperties": {},
+          "type": "object"
+        },
+        {
+          "type": "null"
+        }
+      ],
+      "description": "The job-level scrape settings as stored."
+    },
+    "priority": {
+      "description": "Effective scheduling priority.",
+      "type": "number"
+    },
+    "progress": {
+      "additionalProperties": true,
+      "description": "Live item counters and the credits charged so far.",
+      "properties": {
+        "bytes": {
+          "description": "Total response bytes of finished items.",
+          "type": "number"
+        },
+        "cancelled": {
+          "description": "Items not run because the job was cancelled.",
+          "type": "number"
+        },
+        "completed": {
+          "description": "Items in any terminal status (succeeded + failed + cancelled + skipped).",
+          "type": "number"
+        },
+        "credits_charged": {
+          "description": "Credits actually charged so far (whole credits, rounded down).",
+          "type": "number"
+        },
+        "credits_charged_micro": {
+          "description": "Same as credits_charged in micro-credits (1 credit = 1,000,000).",
+          "type": "number"
+        },
+        "failed": {
+          "description": "Items that failed after all attempts; re-run them with spicrawl_batch_retry.",
+          "type": "number"
+        },
+        "percent_complete": {
+          "description": "completed / total x 100.",
+          "type": "number"
+        },
+        "remaining": {
+          "description": "Items not yet finished (total - completed).",
+          "type": "number"
+        },
+        "skipped": {
+          "description": "Items skipped (for example after failure_threshold or credit_budget stopped the run).",
+          "type": "number"
+        },
+        "succeeded": {
+          "description": "Items that succeeded. Only these are charged.",
+          "type": "number"
+        },
+        "total": {
+          "description": "Items in the job (grows when items are appended to an open job).",
+          "type": "number"
+        }
+      },
+      "type": "object"
+    },
+    "project_id": {
+      "description": "Project the API key belongs to.",
+      "type": "string"
+    },
+    "results_expire_at": {
+      "description": "ISO-8601 deadline after which results answer 410 Gone.",
+      "type": [
+        "string",
+        "null"
+      ]
+    },
+    "results_url": {
+      "description": "API path of the job's results: /v1/batch/{id}/results.",
+      "type": "string"
+    },
+    "started_at": {
+      "description": "ISO-8601 time the first item started.",
+      "type": [
+        "string",
+        "null"
+      ]
+    },
+    "status": {
+      "description": "Lifecycle: queued, running, paused, cancelling, completed, failed or cancelled. Keep polling while queued/running/paused/cancelling; stop at completed, failed or cancelled.",
+      "type": "string"
+    },
+    "status_url": {
+      "description": "API path to poll for this job: /v1/batch/{id}.",
+      "type": "string"
+    },
+    "submitted_at": {
+      "description": "ISO-8601 time the job was accepted.",
+      "type": "string"
+    },
+    "total_items": {
+      "description": "Number of items in the job.",
+      "type": "number"
+    },
+    "warnings": {
+      "anyOf": [
+        {
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        {
+          "type": "null"
+        }
+      ],
+      "description": "Decisions made on the caller's behalf at submission (a clamped concurrency, items not yet queued)."
+    }
+  },
+  "required": [
+    "id"
+  ],
+  "type": "object"
+}
  3. Changed1 schema field changed
    • removedInput schema / properties / engine
      Removed value: -{
-  "description": "Pin the scrape engine for every item: `fetch` or `obscura` (must be allowed by the plan and served by the deployment); `camoufox` is coming soon, do not pin it yet. `chromium` cannot be pinned in a batch — use `render: true`, or spicrawl_scrape.",
-  "enum": [
-    "fetch",
-    "obscura",
-    "camoufox"
-  ],
-  "type": "string"
-}
  4. First observed

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare the write/idempotency/open-world profile, but the description adds genuinely non-obvious behavior: the job runs asynchronously, shared settings cascade to every URL, and an open job "never finishes until you call spicrawl_batch_close" — a lifecycle trap an agent would otherwise hit. It does not cover cost/credit consumption behavior beyond naming `estimated_credits` in the return, so it stops short of full behavioral disclosure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Roughly five sentences, each carrying distinct load — purpose, argument constraint, return shape, workflow routing, alternative, lifecycle — and the most decision-relevant facts (what it does, when to prefer it) are front-loaded. No filler or restated title.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 20-parameter asynchronous job tool with nested items, the description covers the pieces schema alone cannot: the either/or input contract, the async return shape, the sibling chain for polling and retrieval, and the open-job termination rule. With an output schema present, the brief return summary is sufficient rather than redundant.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds a constraint the schema cannot express: with zero required parameters, it states you must supply EITHER `urls` or `items`, "at least one URL, unless `open` is true". That conditional cardinality rule is real meaning beyond the structured fields; the rest of the 20 parameters are left entirely to the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence states a specific verb and resource ("Submit many URLs to be scraped as one asynchronous job") and scopes it with "shared settings applied to every URL", which distinguishes it from the per-URL `spicrawl_scrape` sibling. An agent can identify the tool's job without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It names the explicit alternative and the threshold that selects it ("Use this instead of many `spicrawl_scrape` calls when you have tens or thousands of URLs"), and routes the post-submission workflow to `spicrawl_batch_status`, `spicrawl_batch_results`, and `spicrawl_batch_task_content`. The `open: true` branch points forward to `spicrawl_batch_add_items` and `spicrawl_batch_close`, so no workflow step is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Spicrawl toolsInspect
Try in Browser
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources