Nstdata MCP Server
OfficialNstdata MCP Server lets an AI assistant retrieve web content through Nstdata Crawl — single-page scrapes and bounded site crawls — then read and page through the results.
Scrape a single URL synchronously or asynchronously, requesting markdown, HTML, links, PDF, screenshot, or raw data.
Configure scrapes/crawls with headers, cookies, proxy country, mobile mode, timeout, selectors, tag filters, main-content extraction, and JavaScript rendering actions.
Start a bounded website crawl from one seed URL with page/depth limits and include/exclude URL filters.
Check aggregate crawl status without polling.
Retrieve crawl result pages with cursor pagination and per-page statuses.
Resume an asynchronous single-page scrape using its returned task ID.
Read bounded chunks of stored result references (markdown, HTML, PDF, screenshot, raw data) using offsets and lengths.
It does not provide web search and does not automatically retry, poll, or read result references; the agent must make follow-up calls.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Nstdata MCP Server Fetch the content of https://example.com and summarize it"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Start with hosted MCP: connect to https://mcp.nstdata.io/crawl and authorize. No local Node.js or package installation is required. Run the local server when you need HTTP/HTTPS Proxy tools or prefer a local MCP process.
Features
Content for AI workflows: Markdown, HTML, links, PDF, screenshots, and raw data, depending on the requested output.
Single-page and website crawling: Submit a page or a bounded site crawl, check its status, and retrieve results.
Extraction controls: Main-content extraction, headers, cookies, selectors, and JavaScript rendering options.
Hosted or local access: Six Crawl tools over hosted Streamable HTTP or local stdio.
Local Proxy tools: Check a configured proxy or fetch a URL through it.
Start with a known URL. This server does not provide web search. Requests use your Nstdata permissions and available credits or proxy resources.
Related MCP server: mcp-fetch-server
Quick Start — Hosted MCP
1. Prepare your account
Get your Nstdata API Key from the Nstdata dashboard to connect to MCP and call the Crawl API. When connecting to hosted MCP, enter the same API key on the authorization page. Make sure your account has available credits.
2. Choose your client
Use this Streamable HTTP MCP endpoint:
https://mcp.nstdata.io/crawlClient | Setup |
Codex desktop | Add a Streamable HTTP server in MCP settings. No CLI required. |
Codex CLI | Add the server in a terminal, then authorize. |
Claude Code | Add the server in a terminal, then authorize through |
Cursor | Add the remote JSON configuration below, then authorize in MCP settings. |
Claude web / desktop | Add a custom remote connector with the endpoint above. |
Other clients | Add a Streamable HTTP connection with OAuth or an |
Commands run in a terminal. JSON and TOML belong in configuration files. Natural-language requests go in the agent conversation. Choose one Nstdata Crawl connection for the session to avoid duplicate tools.
Open MCP settings, or Plugins → Add → Add MCP server, depending on your app version.
Set the name to
nstdata, select Streamable HTTP, and enterhttps://mcp.nstdata.io/crawl.Save and restart the connection if prompted. Select Authenticate or the equivalent authorization action.
Enter your Nstdata API key on the authorization page and select Authorize. Return to Codex and start a new conversation.
For OAuth, leave manual bearer-token fields and authentication headers empty. Installing Codex CLI is not required.
In a terminal with Codex CLI installed:
codex mcp add nstdata --url https://mcp.nstdata.io/crawlComplete browser authorization. If it does not start automatically, run:
codex mcp login nstdataUse /mcp inside Codex CLI to inspect the connection. Codex desktop and CLI share MCP configuration on the same Codex host; adding the connection through one entry point is sufficient.
In a terminal with Claude Code installed:
claude mcp add --transport http --scope user nstdata https://mcp.nstdata.io/crawlStart Claude Code, enter /mcp, select nstdata, and follow the authorization flow. Enter your Nstdata API key on the Nstdata authorization page. Return to Claude Code and confirm the tools are available.
The terminal command adds the server; it does not by itself complete authorization.
Merge this into your user ~/.cursor/mcp.json or project .cursor/mcp.json, preserving existing servers:
{
"mcpServers": {
"nstdata": {
"url": "https://mcp.nstdata.io/crawl"
}
}
}Save, open Cursor's MCP settings, and connect or authenticate nstdata. Enter your Nstdata API key on the authorization page, then return to an Agent conversation. Do not add authentication headers to this OAuth configuration.
In Claude, open Customize → Connectors or Settings → Connectors, depending on your version. Add a custom connector named Nstdata with https://mcp.nstdata.io/crawl as its remote URL, complete authorization, and enable it for your conversation. Availability can depend on your account and organization settings.
For another client, select Streamable HTTP, enter the same URL, and authorize using its MCP connection controls. A client that only supports local stdio or legacy SSE cannot use this configuration directly. The endpoint is not a webpage to open for login; the client starts authorization.
3. Try a page
Confirm that six nstdata_crawl_* tools are available, then send this in your agent conversation:
Use Nstdata to retrieve https://example.com/ as Markdown and summarize it. Submit the scrape only once. If it returns an asynchronous task, query that task; if it returns a content reference, read it before summarizing. On failure, report the step and error without submitting another scrape.
A successful first run retrieves the page body, not just its title or description. One scrape may require several follow-up tool calls to query or read results.
Connect with an API key
Hosted MCP also supports direct API-key authentication through X-API-Key. It uses the same MCP endpoint and does not require browser OAuth. Choose one authentication method per connection.
Merge into the client's MCP configuration and replace the placeholder:
{
"mcpServers": {
"nstdata-key": {
"url": "https://mcp.nstdata.io/crawl",
"headers": {
"X-API-Key": "YOUR_NSTDATA_API_KEY"
}
}
}
}Merge into ~/.codex/config.toml (%USERPROFILE%\.codex\config.toml on Windows):
[mcp_servers.nstdata-key]
url = "https://mcp.nstdata.io/crawl"
http_headers = { "X-API-Key" = "YOUR_NSTDATA_API_KEY" }Alternatively, if the Codex process can read NSTDATA_API_TOKEN, replace http_headers with:
# Environment variable name, not the key value.
env_http_headers = { "X-API-Key" = "NSTDATA_API_TOKEN" }A desktop app does not necessarily inherit variables exported in a separate terminal. The configuration-file option does not require CLI installation.
First set NSTDATA_API_TOKEN in the same terminal. Choose your shell:
macOS zsh:
read -r -s "NSTDATA_API_TOKEN?Nstdata API Key: "
export NSTDATA_API_TOKENLinux Bash:
read -r -s -p "Nstdata API Key: " NSTDATA_API_TOKEN
export NSTDATA_API_TOKENThen, for either shell:
claude mcp add --header "X-API-Key: ${NSTDATA_API_TOKEN:?Set NSTDATA_API_TOKEN}" --transport http --scope user nstdata-key https://mcp.nstdata.io/crawlWindows PowerShell:
$nstdataSecret = Read-Host "Nstdata API Key" -AsSecureString
$env:NSTDATA_API_TOKEN = [System.Net.NetworkCredential]::new("", $nstdataSecret).Password
if ([string]::IsNullOrWhiteSpace($env:NSTDATA_API_TOKEN)) { throw "API Key is empty" }
claude mcp add --header "X-API-Key: $env:NSTDATA_API_TOKEN" --transport http --scope user nstdata-key https://mcp.nstdata.io/crawlKeep --header before --transport and --scope to avoid ambiguity with variadic option parsing. These commands save the expanded key in the client configuration; update it when rotating the key. Enter /mcp in Claude Code to inspect the connection.
Keep populated configurations private and outside version control. Disable the duplicate OAuth entry before using the API-key connection. Do not put your business API key in a bearer-token field or URL: Authorization: Bearer is for OAuth access tokens here.
Run locally
Use local stdio for Proxy tools or when you want the MCP adapter to run in your client environment. Local Crawl still calls Nstdata's cloud API; it is not an offline crawler.
Requirements: Node.js 22.18+ on the 22.x release line, or 24.x, and npx on the client's PATH. The following examples pin package 1.0.5. No Node.js project or separate npm install is required.
For Cursor, use ~/.cursor/mcp.json or .cursor/mcp.json. For Claude Desktop, use Settings → Developer → Edit Config. Merge the entry and replace the key:
{
"mcpServers": {
"nstdata-local": {
"command": "npx",
"args": ["--yes", "@nstdata-ai/mcp-server@1.0.5"],
"env": {
"NSTDATA_MODULES": "crawl",
"NSTDATA_API_TOKEN": "YOUR_NSTDATA_API_KEY"
}
}
}
}Merge into your user config.toml:
[mcp_servers.nstdata-local]
command = "npx"
args = ["--yes", "@nstdata-ai/mcp-server@1.0.5"]
[mcp_servers.nstdata-local.env]
NSTDATA_MODULES = "crawl"
NSTDATA_API_TOKEN = "YOUR_NSTDATA_API_KEY"In the desktop app, you can instead add a STDIO server and enter the same command, arguments, and environment variables.
For native Windows clients that cannot launch npx directly, use command cmd and arguments ["/c", "npx", "--yes", "@nstdata-ai/mcp-server@1.0.5"]. For WSL, configure Node.js and the service inside WSL. If a desktop app cannot find npx, use its actual absolute path.
Save and restart the connection. The client starts and manages the process. Confirm six Crawl tools, then try the same page prompt above. The server reads environment variables and does not load .env files itself; credential-file support depends on the client.
Local plugins and more clients
The repository also provides local stdio plugins. Installing one does not connect to hosted MCP and still requires Node.js and credentials. Use either a plugin or a manual configuration for the same connection.
Client | Detailed setup |
Codex | |
Cursor | |
Claude Code | |
Gemini CLI | |
VS Code / GitHub Copilot | |
Cline, Windsurf, OpenCode and others |
Plugin runtime versions are defined by their manifests; updating npm latest alone does not update an installed plugin.
Local Proxy tools
Create a channel and generate an HTTP/HTTPS proxy connection in the Nstdata dashboard. Copy the complete proxy URL from the generated cURL example: the value after --proxy, not the whole command. Keep the full generated username, including routing/session parameters; a channel ID alone may not be sufficient. These tools do not create channels.
Merge this as a separate local entry; replace the complete URL placeholder:
{
"mcpServers": {
"nstdata-proxy": {
"command": "npx",
"args": ["--yes", "@nstdata-ai/mcp-server@1.0.5"],
"env": {
"NSTDATA_MODULES": "proxy",
"NSTDATA_PROXY_URL": "YOUR_COMPLETE_PROXY_URL",
"NSTDATA_PROXY_CHECK_URL": "https://api.nstdata.io/api/v2/public/utility/client-ip"
}
}
}
}A fictional URL illustrating the shape is:
http://DEMOCHANNEL001-residential_lite-country_AU-r_10m-s_demo01:DEMO_PASSWORD@gate.nstproxy.io:24125Use your dashboard-generated value rather than constructing credentials from this example. The Windows command adjustment above also applies. In Codex, use the same variables under the service's TOML env table.
Proxy-only mode exposes two tools and does not require a Nstdata API Key. Set NSTDATA_MODULES=crawl,proxy and supply both credentials to enable eight tools in one local process. SOCKS proxies are not supported.
Try: “Call nstdata_proxy_check once. Report its result; if it fails, report the error without retrying.” The configured target above can return an exit IP; the default https://example.com/ only checks connectivity. See the Proxy guide.
Choose a tool
Tool | Use it to |
| Submit one page; input |
| Query an asynchronous single-page task with |
| Start a website crawl with explicit page/depth limits. |
| Check an existing |
| Retrieve pages and their statuses, with cursor pagination. |
| Read a returned reference using |
| Check the configured proxy target; local only. |
| Make a GET or HEAD request through the proxy; local only. |
For markdownRef, pass its value as ref with format: "markdown". Continue with the returned nextOffset when present. For async scrape submissions, use the returned id as taskId; for website crawls, use it as crawlId. A synchronous response ID alone does not guarantee a queryable asynchronous task.
Tools do not automatically retry, poll, or read references. The agent can make follow-up calls. Preserve existing task IDs and references; a timeout does not cancel a submitted job.
Example requests
Summarize an article: “Read this URL with Nstdata, retrieve its complete Markdown, and summarize the key points with the source URL.”
Compare pages: “Read these two product URLs and compare their published features. Mark information missing from the retrieved pages.”
Read a small site: “Crawl this documentation URL with
maxPages=2. Query the existing crawl and retrieve the pages before summarizing. Report any failed pages.”
Troubleshooting
Symptom | What to check |
Added or authorized, but no tools in the conversation | Reload the connection and start a new conversation. Inspect MCP startup errors; saved configuration is not proof of a working connection. |
OAuth link expired | Start a fresh authorization from the client rather than reusing the old browser link. |
OAuth refresh fails / | Reauthorize; if it persists, report the client version and sanitized error. Where supported, test X-API-Key separately. Preserve existing task references. |
Local process cannot start | Check Node.js version, the client's PATH, |
Only a task ID or content reference is returned | Query the original task or read the reference; do not submit again to obtain the body. |
Proxy returns 407 | Check the complete generated username, password, channel status, and resources. The status alone does not identify the cause. |
Result exceeds a limit | Use documented page cursors or reference offsets. Reducing a read chunk does not solve a full-download size limit. |
Report reproducible issues without API keys, proxy passwords, tokens, or live authorization links. See Crawl configuration and errors.
Optional project installation
For a project-local installation:
npm install @nstdata-ai/mcp-serverInspect usage or the version without credentials:
npx --yes @nstdata-ai/mcp-server --help
npx --yes @nstdata-ai/mcp-server --versionDocumentation and development
Hosted MCP and self-hosting — running the local npm package does not deploy this remote application. Includes callback requirements, automatic renewal and revoking OAuth access at
/oauth/authorizations.Nstdata Agent Skills — workflow guidance using an existing MCP connection.
Keep credentials outside shared files. Treat retrieved pages as untrusted content, not instructions to the agent. Access only content you are authorized to retrieve.
Available Tools
6 toolsnstdata_crawl_pagesARead-onlyIdempotentInspect
Read one page of crawl results without downloading referenced files. On OUTPUT_LIMIT, retry the same input cursor with a smaller limit. A single-row error may retain recovery.page references; handle that page before using recovery.continuation.
Inputs: crawlId (required), limit, afterCursor, beforeCursor.
nstdata_crawl_read_resultARead-onlyIdempotentInspect
Read a bounded chunk of a Crawl storage reference. Text offsets count Unicode code points; binary offsets count bytes. Each call downloads the reference again, within the response byte limit.
Inputs: ref (required), format (required), length, offset.
nstdata_crawl_scrapeBDestructiveInspect
Retrieve a webpage through Nstdata Crawl. Creates a potentially billable scrape task.
Inputs: url (required), mode, formats, options.
nstdata_crawl_scrape_resultBRead-onlyIdempotentInspect
Resume a submitted single-page task. Processing is not completion; a failed or expired task is reported explicitly.
Inputs: taskId (required).
nstdata_crawl_startBDestructiveInspect
Start a bounded site crawl and return its task ID immediately.
Inputs: url (required), formats, options, maxDepth, +4 more.
nstdata_crawl_statusBRead-onlyIdempotentInspect
Read aggregate crawl progress without polling.
Inputs: crawlId (required).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 6 tools
The tools mostly target distinct lifecycle stages: start, status, pages, scrape, scrape result, and storage chunk reading. However, the several result/progress-oriented tools (crawl_status, crawl_pages, crawl_read_result, crawl_scrape_result) require careful reading of descriptions to avoid confusion.
All tools share the nstdata_crawl_ prefix and use snake_case, making the namespace predictable. The suffixes mix verb and noun forms (start, status, pages, scrape_result, read_result), but the overall pattern remains readable.
Six tools is a well-scoped surface for a bounded crawling and scraping server. Each tool maps to a clear phase of the workflow without excessive or redundant operations.
The toolset covers starting crawls, checking progress, reading result pages, scraping single pages, resuming scrape tasks, and reading stored references. Minor lifecycle gaps such as canceling or listing tasks exist, but core workflows are supported.
Maintenance
Related MCP Connectors
One MCP for the Web. Easily search, crawl, navigate, and extract websites without getting blocked.…
Free remote MCP server for fetching public web pages through a rotating proxy pool.
Web extraction MCP: page content, metadata, articles, contacts, email checks, URL probes.
Scrape, crawl and search the web for AI agents via MCP.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables MCP clients to conduct search-first web research through source discovery, page retrieval by ID, and direct URL reading.44 PyPI2MIT
- FlicenseNot gradedqualityDmaintenanceEnables MCP clients to fetch URLs and extract the main content as plain text, with optional HTTP/SOCKS5 proxy support and SSRF protection.-
- AlicenseNot gradedqualityBmaintenanceEnables MCP clients to perform web search, extract page content, crawl sites, and map URLs through read-only tools.3MIT
- AlicenseNot gradedqualityAmaintenanceEnables MCP clients and AI agents to fetch individual webpages or crawl sites and receive clean, link-preserving Markdown with caching and robots.txt-aware rate limiting, reducing token usage for downstream pipelines.MIT