Skip to main content
Glama
guanyuyan
by guanyuyan

web_crawl_start

Start an asynchronous website crawl and immediately return a job ID, allowing multi-page crawls to run in the background without blocking MCP requests.

Instructions

启动一个异步爬取任务,立即返回 job_id;页数多时不再阻塞 MCP 请求。

与 web_crawl 的抓取语义一致,但任务在后台推进、状态落盘到 DATA_DIR/jobs/:

  • 同一组参数重复 start 是幂等的,会返回已有 job_id(existing=true);

  • 进程重启后调用 web_crawl_status 会自动续跑未完成任务;

  • 用 web_crawl_cancel 可随时停止,已完成页面的部分结果仍可读取;

  • 任务状态逐页原子写入,不会因为中途退出而留下半个 JSON。

Args: url: 起始页 URL。 depth: 链接模式的跳转层数(0~3,默认 1)。 limit: 链接模式最多抓取的页面总数(含起始页),受 crawl_max_pages 上限约束。 same_host: 只跟进同主机链接(默认 True)。 max_tokens: 每页正文 token 预算上限。 link_policy: 同 web_fetch。 next_selector: 分页模式:指向"下一页"的 CSS 选择器;有值时忽略 depth。 max_pages: 分页模式最多翻多少页(默认 5),受 crawl_max_pages 上限约束。 deadline_ms: 任务的总预算(毫秒),不传表示不限;超时后不再抓新页。 session: 登录态会话名(默认 default)。 proxy: 本次任务所有页面共用的代理;不传则按代理池分配,未配池时沿用全局配置。

Input Schema

NameRequiredDescriptionDefault
urlYes
depthNo
limitNo
proxyNo
sessionNodefault
max_pagesNo
same_hostNo
max_tokensNo
deadline_msNo
link_policyNointernal
next_selectorNo

Output Schema

NameRequiredDescriptionDefault
resultYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Addedv0.21.0

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full burden and does so: idempotent start returning existing=true, state persisted under DATA_DIR/jobs/, automatic resumption on restart, cancellation preserving partial page results, and per-page atomic writes preventing half-written JSON. These are exactly the non-obvious traits an agent needs before launching a background job.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Behavioral contract is front-loaded before the Args block, and every line of the Args block is load-bearing given zero schema coverage. Slightly long overall, and the single line delegating link_policy to web_fetch is the one entry that doesn't earn its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 11-parameter async job tool with no annotations, the definition covers lifecycle (start/status/cancel), failure/restart semantics, resource limits, and the returned job_id; an output schema exists so return values need no further explanation. Nothing an agent needs to invoke it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the description compensates for all 11 params: depth range 0~3 with default, limit constrained by crawl_max_pages, next_selector switching to pagination mode and overriding depth, deadline_ms as a total budget that stops new pages, and proxy pool fallback behavior. Only link_policy is under-specified ("同 web_fetch").

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb+resource+behavior: starts an asynchronous crawl job and immediately returns a job_id. It explicitly contrasts with the synchronous sibling web_crawl ("与 web_crawl 的抓取语义一致,但任务在后台推进"), so an agent can pick between them without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

States the selecting condition ("页数多时不再阻塞 MCP 请求") and names the lifecycle siblings web_crawl_status and web_crawl_cancel with their roles (resume after restart, stop at any time). It never states the inverse case (small/single-page work → web_crawl or web_fetch), so the routing guidance is clear but not fully closed.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.