Skip to main content
Glama

APIClaw

by io.apiclaw.api

openapi_v2_webtools_crawl_submit

Submit a crawl job

Submit an async recursive crawl job.

Returns an opaque id; poll GET /webtools/crawl/{id} for status and finished pages. robots.txt is always honored; external-domain links are not followed. limit defaults to 100 pages (cap 10000) — set it explicitly when crawling a larger site. The returned id is bound to the calling tenant: polling from a different API key returns 404.

Responses:

200: Successful Response (Success Response) Content-Type: application/json

Example Response:

{
  "success": true,
  "meta": {
    "requestId": "Requestid",
    "timestamp": "Timestamp"
  }
}

Output Schema:

{
  "properties": {
    "success": {
      "type": "boolean",
      "title": "Success",
      "description": "Whether the request was successful",
      "default": true
    },
    "data": {
      "description": "Response data payload"
    },
    "error": {
      "description": "Error details if request failed"
    },
    "meta": {
      "description": "Metadata for API responses.\n\nCredit fields follow the ADR-0003 parallel-fields strategy (Option 3):\n- `credits_remaining` / `credits_consumed` (int): legacy fields, rounded\n  to whole credits, kept for zero-breaking-change to existing SDK clients.\n- `credits_remaining_exact` / `credits_consumed_exact` (float): new\n  precision-aware fields for clients that opt in to decimal credits.\n\nSee ADR-0003 decision 5 and the \u00a78 deprecation timeline.\n\nTODO(2026-11, ADR-0003 \u00a78 +6mo): mark `credits_remaining` /\n`credits_consumed` as `deprecated=True` in their Field() definitions\nand announce in customer changelog.\nTODO(2027-05, ADR-0003 \u00a78 +12mo): remove the legacy int fields via a\nmajor-version bump of the OpenAPI surface.",
      "properties": {
        "requestId": {
          "type": "string",
          "title": "Requestid",
          "description": "Unique request identifier"
        },
        "timestamp": {
          "type": "string",
          "title": "Timestamp",
          "description": "Response timestamp in ISO 8601 format"
        },
        "total": {
          "title": "Total",
          "description": "Total number of records"
        },
        "page": {
          "title": "Page",
          "description": "Current page number"
        },
        "pageSize": {
          "title": "Pagesize",
          "description": "Number of records per page"
        },
        "totalPages": {
          "title": "Totalpages",
          "description": "Total number of pages"
        },
        "creditsRemaining": {
          "title": "Creditsremaining",
          "description": "Remaining API credits (rounded to whole credits; see creditsRemainingExact for precise value)"
        },
        "creditsConsumed": {
          "title": "Creditsconsumed",
          "description": "Credits consumed by this request (rounded; see creditsConsumedExact for precise value)"
        },
        "creditsRemainingExact": {
          "title": "Creditsremainingexact",
          "description": "Remaining API credits, precise to 1 decimal place"
        },
        "creditsConsumedExact": {
          "title": "Creditsconsumedexact",
          "description": "Credits consumed by this request, precise to 1 decimal place"
        },
        "tokensUsage": {
          "description": "Provider token-usage block \u2014 populated on terminal video polls only, null on every non-video endpoint. See TokensUsage for its fields."
        }
      },
      "type": "object",
      "required": [
        "requestId",
        "timestamp"
      ],
      "title": "ResponseMeta"
    }
  },
  "type": "object",
  "required": [
    "meta"
  ],
  "title": "OpenApiResponse[CrawlerCrawlSubmit]",
  "examples": []
}

422: Validation Error Content-Type: application/json

Example Response:

{
  "detail": [
    {
      "loc": [],
      "msg": "Message",
      "type": "Error Type",
      "ctx": {}
    }
  ]
}

Output Schema:

{
  "properties": {
    "detail": {
      "items": {
        "properties": {
          "loc": {
            "items": {},
            "type": "array",
            "title": "Location"
          },
          "msg": {
            "type": "string",
            "title": "Message"
          },
          "type": {
            "type": "string",
            "title": "Error Type"
          },
          "input": {
            "title": "Input"
          },
          "ctx": {
            "type": "object",
            "title": "Context"
          }
        },
        "type": "object",
        "required": [
          "loc",
          "msg",
          "type"
        ],
        "title": "ValidationError"
      },
      "type": "array",
      "title": "Detail"
    }
  },
  "type": "object",
  "title": "HTTPValidationError"
}

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesSeed URL to start crawling from.
limitNoMaximum pages to crawl. Defaults to 100; hard cap 10000.
maxDepthNoMaximum link depth to traverse from the seed.
sitemapModeNoHow the site's sitemap is used for discovery. ``include`` (default) combines the sitemap with link following — the most complete coverage; ``only`` crawls sitemap URLs exclusively (fastest, but misses pages not in the sitemap); ``skip`` ignores the sitemap and discovers by following links only (misses pages that are in the sitemap but not linked, e.g. orphan/deep pages — useful when the sitemap is stale or inaccurate).include
excludePathsNoPath regex blacklist (max 50 patterns, each ≤500 chars).
includePathsNoPath regex whitelist (max 50 patterns, each ≤500 chars).
allowSubdomainsNoFollow links to subdomains of the seed host.
crawlEntireDomainNoCrawl the whole domain rather than only paths nested under the seed URL. By default a seed like ``example.com/blog`` stays under ``/blog``; set ``true`` to also follow sibling/parent paths across the domain.
ignoreQueryParametersNoTreat URLs that differ only by query string as the same page, so they are crawled once. Useful on sites with tracking/pagination params (``?utm=``, ``?page=``) to avoid spending the page budget on near-duplicates.

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses several important behaviors: robots.txt is always honored, external-domain links are not followed, limit defaults to 100 with a cap of 10000, and the returned id is tenant-bound (polling with a different API key returns 404). This is rich, non-obvious behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the essential purpose and key constraints, followed by structured response schemas. It is somewhat lengthy due to the embedded output schemas, but each section serves a reference purpose. There is minor redundancy with the limit information already present in the schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex async crawl tool with 9 parameters, the description covers submission, polling, constraints (robots.txt, external links), pagination limits, tenant binding, and response handling. It explains edge cases and provides example responses, making it comprehensive.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% description coverage; every parameter already includes an explanatory description. The main description repeats the limit default/cap but adds little beyond the schema's own parameter documentation, so the baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Submit a crawl job' and 'Submit an async recursive crawl job,' clearly identifying the verb and resource. It distinguishes from sibling tools by explaining the async nature, the returned opaque id, and the polling endpoint (GET /webtools/crawl/{id}).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear usage context: it is for submitting an async crawl, and explicitly directs the user to poll GET /webtools/crawl/{id} for status. However, it does not explicitly name sibling tools like crawl_status as alternatives, so the guidance is implied rather than fully explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All APIClaw toolsInspect
Try in Browser
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

B3.2/5.0
Disambiguation2/5

Several tools have unclear boundaries, most notably openapi_v2_competitor_product_keywords and openapi_v2_product_traffic_terms, which have identical descriptions. The deprecated openapi_v2_realtime_product duplicates openapi_v3_realtime_product, and openapi_v2_image_embedding overlaps heavily with openapi_v2_fashion_image_embedding.

Naming Consistency2/5

Naming is inconsistent: a few tools follow a clean verb_noun pattern (create_video_asset, list_video_assets, poll_video_task), while the vast majority are prefixed with openapi_v2_/openapi_v3_ followed by nouns or mixed verbs. The route-style prefix is not a meaningful verb and creates an arbitrary split across the tool set.

Tool Count2/5

49 tools is far beyond the well-scoped range and spans multiple unrelated domains (Amazon product data, TikTok commerce, fashion vision, web scraping, video generation, billing). The server appears to be an entire REST API surface exposed wholesale rather than a curated set of capabilities.

Completeness3/5

Each domain is individually fairly complete (Amazon search/reviews/keywords/VoC, TikTok search, web tools, video generation), but there are notable gaps: video assets lack update/delete, video tasks lack cancel/list, and there is no singular 'get daily product by ASIN' alongside the realtime variants. The broad scope makes it hard to verify full lifecycle coverage across all domains.

Resources