Site Crawler for Links and Tables MCP Server

Crawl a website for links and tables

crawl_site_links_tables

Read-onlyIdempotent

Crawl websites over plain HTTP and return one row per page with internal and external links, file download links, HTML tables as rows, and lists; respects robots.txt and JavaScript limits.

Instructions

Crawl a website from a start address over plain HTTP and return one row per page with internal links, external links, file download links, HTML tables as rows, and lists. No AI and no browser, so pages built by JavaScript return little. Obeys robots.txt by default. Pay per event on Apify: 0.006 USD per page.

Input Schema

Name	Required	Description
`extract`	No	Any of links, tables, lists, download_links. Default links, tables, lists. download_links adds one separate row per file link, charged as download-link-row; the download_links field on page rows is always filled at no extra charge.
`max_depth`	No	How many links to follow from a start address. 0 reads only the start address. Default 10.
`max_pages`	No	Pages to read across the whole run, up to 10,000. Default 100.
`proxy_mode`	No	pool (default): the Apify proxy pool only, charged as page-row; a page that refuses it gets a free blocked row. auto: a refused page is retried on residential addresses, and a page read that way is charged as residential-page-row. residential: every page uses residential addresses and is charged as residential-page-row.
`start_urls`	No	Web addresses to start from, up to 100. A bare domain such as example.com is read as https.
`obey_robots`	No	yes (default): skip pages robots.txt disallows and honor its crawl delay up to 5 seconds. no: read them, only if you have the right to.
`use_sitemap`	No	yes (default): also queue the addresses in /sitemap.xml, which finds pages no link reaches. no: follow links only.
`max_charge_usd`	No	Optional. The run stops before a charge that would pass this amount. Summary rows are always returned.
`same_domain_only`	No	yes (default): follow links only on the start address's host and its subdomains. no: also follow links to other sites, up to Maximum pages. Links to other sites are listed either way.
`download_extensions`	No	File extensions that count as download links. Default pdf, zip, xlsx, xls, docx, doc, pptx, csv, mp3, mp4, exe, dmg, msi, pkg, gz, tar, tgz, 7z, rar, iso, apk. Files are listed, never downloaded.

Schema Changelog

Changes observed during successful MCP inspections.

First observedv1.0.0Nov 6, 2026

TDQS

A4.2/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, openWorld, and non-destructive, so the safety profile is covered. The description adds value beyond that with the pricing model ('0.006 USD per page'), the JavaScript rendering limitation, and the default robots.txt obedience — all operational facts an agent must weigh before calling.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four sentences, each carrying distinct load: capability and output shape first, then the rendering caveat, then compliance, then cost. No filler and no repetition of the title.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 10 parameters and no output schema, the description does the right work by describing the row-per-page return and the summary-row behavior implied by max_charge_usd. It does not describe how rows are delivered (dataset vs. inline) or what a blocked/summary row looks like, which would help for a 10-parameter crawl tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter including extract, proxy_mode, and obey_robots is already fully documented in the schema, including defaults and charging implications. The description adds no parameter-level detail beyond the schema, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Crawl a website from a start address') plus the exact return shape ('one row per page with internal links, external links, file download links, HTML tables as rows, and lists'). An agent can tell precisely what it gets back without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a clear negative condition — 'No AI and no browser, so pages built by JavaScript return little' — which tells the agent when not to reach for this tool. No sibling tools exist, so no alternative routing is needed, but it stops short of positive guidance on when this crawl is the right choice (e.g., static sites, link auditing).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools

crawl_site_links_tablesA