Site Crawler for Links and Tables MCP Server
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Site Crawler for Links and Tables MCP ServerCrawl webscraper.io/test-sites/tables and extract all tables as rows"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Site Crawler for Links and Tables MCP Server
An MCP server for the Mamba Labs Site Crawler for Links and Tables actor on Apify. Crawl a website and get its links, download links, tables, and lists as rows.
What's Inside
Related MCP server: Crawl4AI MCP Server
What it does
Give a start address. The actor crawls the site over plain HTTP, following links and the sitemap, and returns one row per page with internal links, external links, file download links, every HTML table as rows, and every list. The output is deterministic, with no AI. robots.txt is obeyed by default. Pages built by JavaScript return little, because no browser runs.
All of the work runs on Apify. This package is a thin client that starts the actor run, waits for it, and hands back the dataset rows.
Quick start
You need Node.js 18 or newer and an Apify account with an API token.
Add this to your Claude Desktop config (claude_desktop_config.json), Cursor, or any MCP client:
{
"mcpServers": {
"mamba-site-crawler-links-tables": {
"command": "npx",
"args": [
"-y",
"@mambalabsdev/mcp-site-crawler-links-tables"
],
"env": {
"APIFY_TOKEN": "your-apify-token"
}
}
}
}Restart the client. The crawl_site_links_tables tool appears.
Prerequisites
Node.js 18 or newer.
An Apify API token from console.apify.com/account/integrations, set as
APIFY_TOKEN. The server lists its tool without a token; a call needs one.
Example prompts
"List every external download link on this website."
"Pull all the tables from these pages into rows."
"Crawl this site and show me which pages link out to other domains."
Tool
crawl_site_links_tables. One tool. Example arguments:
{
"start_urls": [
"https://webscraper.io/test-sites/tables"
],
"extract": [
"tables",
"links"
],
"max_pages": 10,
"max_depth": 1
}Inputs
Every buyer facing input of the actor. Omitted inputs take the actor's own defaults.
Input | Type | Required | Description |
| array | no | Web addresses to start from, up to 100. A bare domain such as example.com is read as https. |
| number | no | Pages to read across the whole run, up to 10,000. Default 100. |
| number | no | How many links to follow from a start address. 0 reads only the start address. Default 10. |
| string | no | yes (default): follow links only on the start address's host and its subdomains. no: also follow links to other sites, up to Maximum pages. Links to other sites are listed either way. |
| array | no | Any of links, tables, lists, download_links. Default links, tables, lists. download_links adds one separate row per file link, charged as download-link-row; the download_links field on page rows is always filled at no extra charge. |
| array | no | File extensions that count as download links. Default pdf, zip, xlsx, xls, docx, doc, pptx, csv, mp3, mp4, exe, dmg, msi, pkg, gz, tar, tgz, 7z, rar, iso, apk. Files are listed, never downloaded. |
| string | no | yes (default): also queue the addresses in /sitemap.xml, which finds pages no link reaches. no: follow links only. |
| string | no | yes (default): skip pages robots.txt disallows and honor its crawl delay up to 5 seconds. no: read them, only if you have the right to. |
| string | no | Optional. The run stops before a charge that would pass this amount. Summary rows are always returned. |
| string | no | pool (default): the Apify proxy pool only, charged as page-row; a page that refuses it gets a free blocked row. auto: a refused page is retried on residential addresses, and a page read that way is charged as residential-page-row. residential: every page uses residential addresses and is charged as residential-page-row. |
Pricing
Pay per event on Apify, billed to your Apify account. Prices in USD.
Event | Free plan | Bronze | Silver | Gold |
| $0.006 | $0.0057 | $0.0054 | $0.0051 |
| $0.02 | $0.019 | $0.018 | $0.017 |
| $0.0001 | $0.000095 | $0.00009 | $0.000085 |
| $0.0055 | $0.0055 | $0.0055 | $0.0055 |
Failed or refused pages, summary rows, pages skipped by robots.txt, and refused input are free. Mamba Labs pays the platform costs of the run, Apify proxy traffic included. Mamba Labs pays the platform costs of the run, Apify proxy traffic included.
Full actor documentation
apify.com/mambalabs/site-crawler-links-tables
License
MIT. See LICENSE.
Available Tools
1 toolcrawl_site_links_tablesCrawl a website for links and tablesARead-onlyIdempotent
Crawl a website from a start address over plain HTTP and return one row per page with internal links, external links, file download links, HTML tables as rows, and lists. No AI and no browser, so pages built by JavaScript return little. Obeys robots.txt by default. Pay per event on Apify: 0.006 USD per page.
| Name | Required | Description | Default |
|---|---|---|---|
| extract | No | Any of links, tables, lists, download_links. Default links, tables, lists. download_links adds one separate row per file link, charged as download-link-row; the download_links field on page rows is always filled at no extra charge. | |
| max_depth | No | How many links to follow from a start address. 0 reads only the start address. Default 10. | |
| max_pages | No | Pages to read across the whole run, up to 10,000. Default 100. | |
| proxy_mode | No | pool (default): the Apify proxy pool only, charged as page-row; a page that refuses it gets a free blocked row. auto: a refused page is retried on residential addresses, and a page read that way is charged as residential-page-row. residential: every page uses residential addresses and is charged as residential-page-row. | |
| start_urls | No | Web addresses to start from, up to 100. A bare domain such as example.com is read as https. | |
| obey_robots | No | yes (default): skip pages robots.txt disallows and honor its crawl delay up to 5 seconds. no: read them, only if you have the right to. | |
| use_sitemap | No | yes (default): also queue the addresses in /sitemap.xml, which finds pages no link reaches. no: follow links only. | |
| max_charge_usd | No | Optional. The run stops before a charge that would pass this amount. Summary rows are always returned. | |
| same_domain_only | No | yes (default): follow links only on the start address's host and its subdomains. no: also follow links to other sites, up to Maximum pages. Links to other sites are listed either way. | |
| download_extensions | No | File extensions that count as download links. Default pdf, zip, xlsx, xls, docx, doc, pptx, csv, mp3, mp4, exe, dmg, msi, pkg, gz, tar, tgz, 7z, rar, iso, apk. Files are listed, never downloaded. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, openWorld, and non-destructive, so the safety profile is covered. The description adds value beyond that with the pricing model ('0.006 USD per page'), the JavaScript rendering limitation, and the default robots.txt obedience — all operational facts an agent must weigh before calling.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four sentences, each carrying distinct load: capability and output shape first, then the rendering caveat, then compliance, then cost. No filler and no repetition of the title.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 10 parameters and no output schema, the description does the right work by describing the row-per-page return and the summary-row behavior implied by max_charge_usd. It does not describe how rows are delivered (dataset vs. inline) or what a blocked/summary row looks like, which would help for a 10-parameter crawl tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so every parameter including extract, proxy_mode, and obey_robots is already fully documented in the schema, including defaults and charging implications. The description adds no parameter-level detail beyond the schema, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Crawl a website from a start address') plus the exact return shape ('one row per page with internal links, external links, file download links, HTML tables as rows, and lists'). An agent can tell precisely what it gets back without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a clear negative condition — 'No AI and no browser, so pages built by JavaScript return little' — which tells the agent when not to reach for this tool. No sibling tools exist, so no alternative routing is needed, but it stops short of positive guidance on when this crawl is the right choice (e.g., static sites, link auditing).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
1 tool update
v1.0.0- First observed
crawl_site_links_tables
TDQS
Scored across 1 tool
There is only one tool, so there is no possible overlap or misselection risk. Its purpose (crawl a site and return links, tables, and lists) is stated explicitly and leaves no ambiguity about when to use it.
The single name crawl_site_links_tables follows snake_case with a leading verb, which is a predictable convention. It is descriptive but concatenates multiple output types into one long name, which is slightly less clean than a tighter verb_noun form.
One tool is thin for a server, though this tool is substantial and bundles several extraction outputs rather than being trivial. A single entry point is workable for a focused crawler, but splitting configuration or output modes would make the surface easier to target.
The tool covers the stated domain well: internal/external/file links, tables, and lists, with robots.txt compliance noted. Gaps are minor-to-moderate: no depth or max-page control surfaced, no JS rendering fallback, and no way to isolate a single page versus a full crawl.
Maintenance
Related MCP Connectors
Scrape and crawl websites into queryable tables: products, leads, Shopify catalogs, real estate.
Web scraping, SERP search, crawl/map and 74 Collectors through residential proxies. 25 tools.
Extract web pages, crawl and map sites, search the web. Extract needs no key; other tools need one.
Scrape websites by following links and extract structured data from natural language instructions.
Related MCP Servers
- AlicenseAqualityDmaintenanceEnables web content scanning and analysis by fetching, analyzing, and extracting information from web pages using tools like page fetching, link extraction, site crawling, and more.613MIT
- FlicenseNot gradedqualityDmaintenanceEnables advanced web crawling and content extraction with JavaScript support, AI-powered analysis, PDF/Office document processing, YouTube transcript extraction, Google search integration, and multi-format data export capabilities.2-
- AlicenseNot gradedqualityCmaintenanceEnables web crawling and content extraction from web pages, supporting multiple output formats like text, markdown, XML, and JSON, with robots.txt compliance and rate limiting.14 npm1MIT
- FlicenseAqualityCmaintenanceEnables fetching, crawling, and analyzing web pages with 29 tools for SEO audits, content extraction, and more.29-