Which integrations are available for this server?

Searches arXiv for academic papers. Searches the web using DuckDuckGo and returns results. Searches GitHub repositories. Searches MDN Web Docs for web development documentation. Looks up package versions from npm (Node Package Manager). Looks up package versions from NuGet (.NET). Looks up package versions from Packagist (PHP Composer). Searches PubMed for biomedical literature. Looks up package versions from PyPI (Python Package Index). Searches Reddit for posts and discussions. Looks up package versions from RubyGems (Ruby). Searches via a self-hosted SearXNG instance. Searches the Stack Exchange network (including Stack Overflow) for questions and answers. Looks up package versions for Swift packages via GitHub git tags. Searches Wikidata entities. Searches Wikipedia for articles and returns relevant results.

How do I use solocrawl?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@solocrawl search for python async programming tips" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

solocrawl

by hlavacm

Overview Schema Source Related Servers Discussions Admin

Python

Local

SoloCrawl is a small, self-hosted, fully async Python tool that does three things well: it searches the web across many sources at once, scrapes pages into clean markdown that's ready for an LLM, and looks up the latest package version from official registries. It runs on your machine, for free, and works the moment you install it — nothing to sign up for, no keys to paste.

It's built for individual developers and people tinkering with local LLMs. Use it as an MCP server (LM Studio, Claude Desktop, OpenCode, …), straight from the CLI, or as a Python library.

✨ Highlights


🔌 Zero config	Search, scrape, and package lookup all work out of the box — no accounts, no keys.
🔎 Federated search	Queries 11 possible sources, merges them with Reciprocal Rank Fusion and de-duplicates URLs into one clean ranking.
📄 Smart scraping	HTML → tidy markdown via `trafilatura`/`readability`, with a Playwright browser fallback for JS-heavy pages — used only when it's actually needed.
📦 Live package versions	10 ecosystems (PyPI, npm, crates.io, Maven, Go, …) resolved live from official registries — never a stale local DB.
🤖 MCP-native	Drops straight into local LLM tooling as a stdio MCP server with five ready tools.
⚡ Fully async, bounded	One shared HTTP client, one recycled browser, global + per-domain concurrency limits. Fast without hammering anyone.
🧩 Hackable	Add a search or package provider as a single self-registering file — the core stays untouched.
🔒 Safe by default	Blocks localhost/private/cloud-metadata targets and honours `robots.txt`.

Related MCP server: mcp-internet

🚀 Quick start

The recommended way to install SoloCrawl is pipx — it drops the solocrawl and solocrawl-mcp commands onto your PATH in their own isolated environment, so you can run them from anywhere without juggling a virtualenv:

git clone https://github.com/hlavacm/solocrawl.git
pipx install ./solocrawl     # or an absolute path: pipx install /path/to/solocrawl

Already on PyPI? Then it's just pipx install solocrawl — no checkout needed.

That's it — now run the three core commands from any directory:

# Scrape a page to markdown
solocrawl scrape https://example.com

# Federated web search (Wikipedia + DuckDuckGo + StackExchange by default)
solocrawl search "python asyncio semaphore" --limit 5

# Live package version lookup
solocrawl package requests --ecosystem pypi

Updating

# Installed from PyPI:
pipx upgrade solocrawl

# Installed from a local checkout — pull the latest changes, then reinstall:
cd /path/to/solocrawl && git pull && pipx install --force .   # alias: pipx reinstall solocrawl

After upgrading, restart your MCP client (LM Studio, Cursor, Claude Desktop, …) so it picks up the new solocrawl-mcp binary — your mcp.json needs no changes as long as it points at solocrawl-mcp on your PATH.

What you can do

🔎 Search the web

One query, many sources, a single merged ranking — no single provider deciding everything for you.

solocrawl search "python asyncio semaphore" --limit 5

# Pick exactly which sources to hit, and get machine-readable output
solocrawl search "django orm" --sources wikipedia,stackexchange --json

📄 Scrape a page to clean markdown

Turn any URL into LLM-ready markdown with page metadata (title, author, date, …) as front-matter.

solocrawl scrape https://example.com

# Save to a file, or force the browser for a JS-rendered page
solocrawl scrape https://example.com --out page.md
solocrawl scrape https://example.com --force-browser

📦 Look up package versions

The current version — and the one matching your constraint — straight from the official registry.

solocrawl package react --ecosystem npm --constraint ">=18,<19"
solocrawl package monolog/monolog --ecosystem packagist --json
solocrawl package some-lib --ecosystem pypi --allow-prerelease

🧪 Research in one shot

The classic LLM workflow — search, scrape the top hits, and get back one aggregated, cited report.

solocrawl research "python asyncio semaphore" --depth 3

🗂️ Batch-scrape many URLs

Fetch a whole list at once under the same bounded concurrency; --out-dir writes one file per URL.

solocrawl batch https://example.com https://www.python.org --out-dir /tmp/scrape
solocrawl batch --from-file urls.txt --out-dir /tmp/scrape

…and see what's available

# List every registered provider (search + package), default vs. opt-in
solocrawl providers

🤖 Use it with your local LLM (MCP)

This is where SoloCrawl really shines — give your local model (LM Studio, OpenCode, Claude Desktop, …) the ability to search, scrape, and check versions. The pipx install from the Quick start already put solocrawl-mcp on your PATH, so all that's left is pointing your MCP client at it.

LM Studio / Claude Desktop — ready-to-use config at examples/mcp.json. Drop it into your client's MCP settings (mcp.json):

{
  "mcpServers": {
    "solocrawl": {
      "command": "solocrawl-mcp",
      "args": [],
      "env": {
        "SOLOCRAWL_LOG_LEVEL": "INFO",
        "SOLOCRAWL_LOG_FILE": "~/.local/state/solocrawl/mcp.log"
      }
    }
  }
}

OpenCode — uses a different config format. Copy examples/opencode.jsonc into ~/.config/opencode/opencode.jsonc (global) or opencode.jsonc in your project root. OpenCode expects type: "local", command as an array, and environment instead of env.

If your MCP client doesn't inherit your shell PATH, replace "solocrawl-mcp" with the full path from which solocrawl-mcp (typically ~/.local/bin/solocrawl-mcp after pipx install). Logs go to stderr (visible in LM Studio Developer Logs) and optionally to SOLOCRAWL_LOG_FILE.

The server exposes five tools:

web_search(query, limit=5, sources=None) — federated search across enabled providers
scrape(url) — fetch and extract markdown (with page metadata) from a URL
research(query, depth=3) — search, scrape the top results, and return an aggregated cited report
package_version(name, ecosystem, constraint=None, allow_prerelease=False) — live registry lookup
list_providers(provider_type="all") — list registered search/package providers (default vs. opt-in)

To check the active version and command path:

pipx list | grep solocrawl
which solocrawl-mcp

Working from a local clone? A pipx-installed solocrawl-mcp is a snapshot — editing the repo does not update the command on your PATH, so your MCP client keeps running the old code. After changing the source, refresh it with pipx install --force . (or install once with pipx install --editable . so future edits are picked up automatically).

🐍 Use it from Python

import asyncio

from solocrawl.config import load_config
from solocrawl.core.search import federated_search, select_providers
from solocrawl.core.search.providers import duckduckgo, stackexchange, wikipedia  # noqa: F401


async def main() -> None:
    providers = select_providers(load_config())
    results = await federated_search(providers, "asyncio python", limit=3)
    for result in results:
        print(result.title, result.url)


asyncio.run(main())

See examples/library_search.py for a runnable example.

Search providers

Default (zero-config, always enabled):

Provider	Source
`wikipedia`	MediaWiki API
`duckduckgo`	`ddgs` package
`stackexchange`	Stack Exchange API (Stack Overflow)

Opt-in (enable with SOLOCRAWL_ENABLE_PROVIDERS):

Provider	Source
`wikidata`	Wikidata entity search
`hackernews`	Hacker News (Algolia)
`arxiv`	arXiv Atom API
`pubmed`	PubMed/NCBI E-utilities
`github`	GitHub repository search
`mdn`	MDN Web Docs search
`searxng`	Self-hosted SearXNG (set `SOLOCRAWL_SEARXNG_URL`)

The Reddit adapter was removed because Reddit requires approved OAuth access. Legacy SOLOCRAWL_ENABLE_PROVIDERS=reddit settings no longer activate a provider.

SOLOCRAWL_ENABLE_PROVIDERS=arxiv,hackernews solocrawl search "transformer attention" --limit 6
SOLOCRAWL_ENABLE_PROVIDERS=github,mdn solocrawl search "fetch api" --limit 6

Package registries

Default ecosystems: PyPI, npm, Packagist, crates.io, NuGet, Maven Central, RubyGems, Go modules, pub.dev, Swift. Versions are always fetched live from official registries — SoloCrawl does not maintain its own version database. Swift packages have no central registry, so versions come from the repository's git tags (owner/repo on GitHub).

solocrawl package serde --ecosystem crates
solocrawl package Newtonsoft.Json --ecosystem nuget
solocrawl package org.junit.jupiter:junit-jupiter --ecosystem maven
solocrawl package github.com/gorilla/mux --ecosystem go
solocrawl package apple/swift-argument-parser --ecosystem swift

Version constraints

PyPI uses PEP 440 (>=1.2,<2, ~=1.4, ==1.2.*). Other registries use a common SemVer range subset: comparators, exact versions, ^, ~, partials and wildcards, whitespace/comma AND, || OR and hyphen ranges (1.2 - 1.3 includes the whole 1.3 series). SemVer prereleases such as canary.1 retain their ordering and bounds; enable them with --allow-prerelease. Native Maven/NuGet interval notation and RubyGems ~> are not supported and are rejected clearly. SoloCrawl does not claim to implement each registry's complete native range grammar. Maven, NuGet and RubyGems retain numeric/standard prerelease ordering through packaging, including four-part numeric releases; SemVer ecosystems use semver. PyPI releases without files or with all files yanked are excluded; mixed yanking keeps the release available. Swift follows GitHub tag pagination, and Go module paths use the proxy's case escaping.

Optional extras

# Browser fallback for JS-heavy pages (Playwright)
pip install -e ".[browser]"
playwright install chromium

solocrawl scrape https://example.com --force-browser

# Install everything
pip install -e ".[all]"

⚙️ Configuration

All defaults work with no configuration. Everything below is optional and uses the SOLOCRAWL_ prefix. For local development, copy .env.dist to .env and uncomment what you need — SoloCrawl loads .env automatically via python-dotenv, and existing shell environment variables take precedence.

Variable	Default	Purpose
`SOLOCRAWL_ENABLE_PROVIDERS`	(empty)	Comma-separated opt-in provider names
`SOLOCRAWL_SEARXNG_URL`	(empty)	Base URL of a self-hosted SearXNG instance (enables the `searxng` provider)
`SOLOCRAWL_RESPECT_ROBOTS`	`true`	Honour `robots.txt` on scrape (fail-open); set `false` to skip
`SOLOCRAWL_CACHE_TTL_SECONDS`	`0`	In-memory fetch cache TTL in seconds (`0` = disabled)
`SOLOCRAWL_MAX_CONCURRENCY`	`10`	Global fetch concurrency limit
`SOLOCRAWL_PER_DOMAIN_LIMIT`	`2`	Per-domain concurrency limit
`SOLOCRAWL_TIMEOUT_SECONDS`	`30`	Per-request timeout in seconds
`SOLOCRAWL_MAX_RETRIES`	`3`	Retries on network errors / rate limits
`SOLOCRAWL_MAX_RESPONSE_BYTES`	`10485760`	Cap on fetched response body size (10 MiB); larger bodies are truncated
`SOLOCRAWL_PROXY_ENABLED`	`false`	Enable optional proxy layer
`SOLOCRAWL_PROXY_MODE`	`list`	Proxy mode: `list` (rotate a pool) or `endpoint` (single rotating endpoint)
`SOLOCRAWL_PROXY_LIST`	(empty)	Comma-separated proxy URLs
`SOLOCRAWL_PROXY_ENDPOINT`	(empty)	Single rotating proxy endpoint
`SOLOCRAWL_PROXY_USERNAME`	(empty)	Proxy auth username
`SOLOCRAWL_PROXY_PASSWORD`	(empty)	Proxy auth password
`SOLOCRAWL_ALLOW_INTERNAL_URLS`	`false`	Allow scraping localhost/private IPs (dev only)
`SOLOCRAWL_USER_AGENT`	(SoloCrawl default)	Override HTTP User-Agent for API requests
`SOLOCRAWL_BROWSER_ALLOWED`	`true`	Allow Playwright fallback when installed
`SOLOCRAWL_LOG_LEVEL`	`WARNING`	Log level: `DEBUG`, `INFO`, `WARNING`, `ERROR`
`SOLOCRAWL_LOG_FILE`	(empty)	Optional log file path (also logs to stderr)

Concurrency, response-size limits and timeout must be positive; retries and cache TTL must be non-negative. Timeout must be finite. Invalid settings report the environment-variable name rather than hanging a request. Catch proxy exhaustion in library code with from solocrawl.core.proxy import ProxyUnavailableError. Enabling a proxy requires valid HTTP(S) endpoints. A depleted proxy pool raises ProxyUnavailableError; it never silently switches to a direct connection. An endpoint that fails to connect is skipped for 60 seconds and then tried again; a 429/503 from the target site does not count against the proxy. Robots checks and browser requests use the selected proxy too.

🔒 Security note on URL fetching

By default SoloCrawl refuses to fetch localhost, link-local, private, reserved, shared (CGNAT/Tailscale 100.64.0.0/10) and other non-global, and cloud-metadata addresses. It checks literal hosts, DNS-resolved A/AAAA records, HTTP redirect targets, and every routed browser request (including frames and subresources). Browser redirects are fetched one hop at a time, service workers and WebSockets are blocked, and the final browser URL is checked. SoloCrawl is still a single-user local tool, not a hostile-multi-tenant proxy — do not expose it to untrusted network callers. SOLOCRAWL_ALLOW_INTERNAL_URLS=true disables these internal-target checks entirely (intended for trusted local development only).

DNS checks and the eventual connection use separate resolution steps, so they do not eliminate DNS rebinding or a proxy resolving a hostname differently. Use network-level egress controls when strong isolation is needed. robots.txt is enforced on every HTTP redirect destination and browser request, with the existing fail-open policy for unavailable or redirected robots files.

Browser main-page redirects retain their final URL. Redirected iframe, script and XHR subrequests are aborted after validating the destination: replaying them through Playwright's fetch API would change browser origin/relative-URL semantics and could forward authorization or POST bodies. This restriction may prevent some pages from rendering completely. Ordinary HTTP redirects still work.

🧩 Extending it

The whole point of the plugin layout is that adding a source is a single self-registering file — the core never changes. To add a search provider:

Create src/solocrawl/core/search/providers/myprovider.py implementing SearchProvider.
Register with @register("myprovider", zero_config=True, configurable=True) or as opt-in. Configurable providers accept __init__(self, *, config: Config | None = None); selectors pass the supplied configuration. Legacy zero-argument providers keep configurable=False (the default).
Import the module in src/solocrawl/core/search/providers/__init__.py so registration runs.
Add fixture-based tests in tests/.

The same pattern applies to package providers under src/solocrawl/core/packages/providers/.

Development

Work from a checkout in a virtualenv with an editable install — this also drops the solocrawl and solocrawl-mcp scripts into .venv/bin/:

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

The MCP server uses FastMCP 4 / MCP SDK v2. Its five tools and stdio wire format stay the same; client-side Python protocol fields use snake_case (input_schema, is_error). See the official migration guide.

Then run the quality gate:

ruff check . && ruff format --check .
pyright
pytest

# …or all in one line:
ruff check . && pyright && pytest

For guided verification and shareable JSON feedback, see docs/verification.md:

python scripts/verify_project.py --mode offline
python scripts/verify_project.py --mode browser
python scripts/verify_project.py --mode all

Live tests are optional. The normal suite excludes live and real-browser tests; CI runs local Chromium fixtures separately. Release checks are listed in docs/release/CHECKLIST.md.

Ethics and terms of use

SoloCrawl is built for individual developers and local LLM tooling. It respects the robots.txt and terms of service of target sites — scrape consults robots.txt and refuses disallowed URLs by default (fail-open on errors; opt out with SOLOCRAWL_RESPECT_ROBOTS=false). The proxy and scraping features are not intended to bypass site rules, captchas, or anti-bot systems. Use responsibly and stay within legitimate access patterns.

License

MIT — see LICENSE.

Available Tools

5 tools

list_providersList ProvidersB

List the registered search and package providers (default vs. opt-in).

ParametersJSON Schema

Name	Required	Description	Default
`provider_type`	No	Which providers to list: 'search', 'package', or 'all'.	all

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

B3.4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. 'List' implies a read-only operation and the default/opt-in distinction adds useful context, but it does not explicitly state safety, auth requirements, or pagination behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, front-loaded sentence with zero waste. It states the action and scope immediately.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple read-only list tool with full parameter schema coverage and an output schema, the description is mostly complete. It states what is listed and the default/opt-in distinction, though it could say more about when to use this versus siblings.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the sole provider_type parameter is already documented. The description mentions search and package providers and default vs. opt-in, but adds no parameter syntax or value meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states a specific verb (List) and resource (registered search and package providers), adding a meaningful scope distinction between default and opt-in providers. It does not explicitly differentiate itself from siblings such as web_search or package_version, but the resource is distinct enough for selection.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No when-to-use or when-not-to-use guidance is given, and no alternatives or prerequisites are mentioned. An agent can infer it is for inspecting provider registration, but the description does not route between this and sibling tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

package_versionPackage VersionB

Look up the latest or constraint-satisfying version of a package from a registry.

ParametersJSON Schema

Name	Required	Description
`name`	Yes	Package name (Packagist: vendor/package, Maven: groupId:artifactId, Go: module path, Swift: owner/repo on GitHub).
`ecosystem`	Yes	Registry ecosystem (pypi, npm, packagist, crates, nuget, maven, rubygems, go, pub, swift).
`constraint`	No	Optional version constraint such as '>=4.2,<5'.
`allow_prerelease`	No	Whether pre-release versions may be selected.

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

B3.4/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. 'Look up' strongly implies a read-only operation, but the description says nothing about auth requirements, registry/network failure modes, rate limits, or what happens when no version satisfies the constraint. For a tool with zero annotation coverage, these gaps are significant.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with zero filler. Every clause ('latest or constraint-satisfying', 'from a registry') contributes meaning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need no explanation, and the schema fully documents the four parameters including per-ecosystem name syntax. The only remaining gap is behavioral context around failure and prerelease handling, which keeps it from a 5.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so name, ecosystem, constraint, and allow_prerelease are all documented in the schema, including ecosystem-specific name formats. The description's 'constraint-satisfying' echoes the constraint parameter but adds no format or edge-case detail beyond what the schema already provides. Baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'look up ... version of a package from a registry.' This is unambiguous and clearly distinct from unrelated siblings like web_search, scrape, and list_providers. It stops short of naming any sibling or contrasting scope, so it lands at 4 rather than 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'latest or constraint-satisfying' implies two usage modes, but the description never states when to use this tool versus web_search or research, nor any prerequisites. Usage is inferable from the name and description alone, which is the definition of a 3.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

researchResearchB

Search the web, scrape the top results, and return an aggregated cited report.

ParametersJSON Schema

Name	Required	Description	Default
`depth`	No	How many top results to scrape and aggregate (default 3).
`query`	Yes	The research query.

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

B3.4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must carry the full behavioral burden. It usefully discloses the pipeline and the cited-report output, but omits operational details such as latency, rate limits, authentication needs, or failure modes for a web-dependent tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single front-loaded sentence with no wasted words. It efficiently communicates the full action sequence and output in one line.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given only two parameters, full schema coverage, and an output schema, the description is complete enough to invoke the tool correctly. The main missing element is usage context relative to sibling tools, but the core behavior is adequately covered.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both query and depth are already documented. The description adds only the idea of 'top results' and does not provide syntax or format details beyond what the input schema already supplies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states specific verbs and resource actions: search the web, scrape top results, and return an aggregated cited report. This distinguishes it in substance from sibling tools like web_search and scrape, although it does not name those siblings explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies a multi-step research workflow but gives no explicit guidance on when to use this tool versus web_search or scrape. There are no stated conditions, prerequisites, or exclusions to help an agent choose among alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrapeScrapeA

Fetch a URL and return the main page content as markdown suitable for LLM context.

ParametersJSON Schema

Name	Required	Description	Default
`url`	Yes	The HTTP or HTTPS URL to scrape.

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

A3.6/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden. It does disclose the salient trait that it extracts 'main page content' into markdown rather than returning raw HTML, which is useful. It says nothing about permissions, rate limits, timeouts, JavaScript-rendered pages, or failure behavior on non-HTML responses.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One sentence, front-loaded with the action and ending with the payoff. Every clause earns its place and nothing is padded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be restated, and this is a one-parameter tool with a low complexity profile. The description covers purpose and output shape adequately; the only real gap is guidance against the sibling search/research tools and any caveats about dynamic content.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% for the single url parameter, which already states it must be an HTTP or HTTPS URL. The description adds no format, normalization, or constraint detail beyond the schema, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description gives a specific verb (fetch), a specific resource (a URL), and the exact output form (main page content as markdown). An agent can immediately tell this retrieves a known page rather than searching. It stops short of naming or contrasting the closest sibling, web_search, so a 5 is not warranted.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: the tool fetches a URL you already have, and the 'suitable for LLM context' framing hints at its role. There is no explicit when-to-use, when-not-to-use, or comparison against web_search or research, which are the obvious alternatives an agent would weigh.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_searchWeb SearchB

Search the web across SoloCrawl's configured providers and return unified results.

ParametersJSON Schema

Name	Required	Description
`limit`	No	Maximum number of merged results to return.
`query`	Yes	The search query.
`sources`	No	Optional provider names to restrict the search (e.g. wikipedia, duckduckgo).

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

B3.3/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It does disclose the meaningful trait that results are merged from multiple configured providers into a unified list, which the schema does not convey. However, it says nothing about provider failures, rate limits, deduplication, or auth requirements, leaving notable gaps for a no-annotation tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with no filler; the aggregation behavior is stated first and nothing is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be explained, and the three parameters are simple. Still, the description omits how to choose this tool over 'research'/'scrape' and what happens when an unknown or unavailable provider is passed in 'sources', leaving it only minimally complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents 'query', 'limit', and 'sources' (including provider examples). The description adds no syntax, format, or behavioral detail beyond that, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Search the web') plus the aggregation scope ('across SoloCrawl's configured providers and return unified results'), which tells the agent this is a federated search rather than a single-provider one. It does not explicitly distinguish itself from the 'research' or 'list_providers' siblings, so it falls short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use guidance and no routing to alternatives: the agent is not told when to prefer this over 'research' (deeper multi-step investigation) or 'scrape' (single-page fetch). Usage is only implied by the tool name.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

5 tool updatesv1.0.0Jun 27, 2026
- First observedlist_providers
- First observedpackage_version
- First observedresearch
- First observedscrape
- First observedweb_search

Deploy Server

Let your AI agent see the webWeb-to-screenshot MCP for Claude, Cursor and Codex. No cookie banners. Free to start.snap-render.comAd

TDQS

A3.6/5.0

Scored 2026-10-23 05:30 across 5 tools

Disambiguation4/5

web_search, scrape, and research have overlapping search/scrape functionality, but their descriptions clearly differentiate outputs (raw results vs. single-page markdown vs. aggregated cited report). package_version and list_providers are unambiguously distinct, so only minor misselection risk exists.

Naming Consistency3/5

All names use lowercase snake_case for multiword terms, but grammatical patterns vary: scrape/research are bare verbs, package_version is noun_noun, web_search is noun_verb, and list_providers is verb_noun. The mix is readable but lacks a predictable convention.

Tool Count5/5

Five tools is well-scoped for a web research and crawling server. Each tool covers a distinct operation (search, scrape, aggregate research, package lookup, provider listing) without redundancy or bloat.

Completeness4/5

Core search, scrape, and research workflows are covered, and package lookup is a useful addition. However, a server named solocrawl lacks a multi-page crawl or batch scrape tool, and provider configuration is absent, leaving minor gaps.

Maintenance

ActivityMaintained

ResponsivenessNo issues

Resources

GitHub Repository

Need Help?

Related Servers

Related MCP Connectors

glim.sh A
sh.glim
Give your agent live data from Twitter, Reddit, the web and GitHub. No API keys, no scraping stack.
Browser Automation Social Media Version Control
1434
agentsearch-mcp A
io.github.IsaiahDupree
LLM-ready web search + instant answers + URL-to-clean-text fetch for agents and RAG.
Search RAG Systems Browser Automation
31
Levirge SearchOAuth
com.levirge
Web search and page reading for AI agents, including JSON APIs, RSS feeds and YouTube.
Research & Data Search
1
Ujeebu B
com.ujeebu
Web scraping, Google SERP, Markdown and ChatGPT/Gemini answers as typed tools for AI agents.
Browser Automation Web Scraping AI & Machine Learning
171

Related MCP Servers

WebFetch.MCP
manooll
A
license
A
quality
C
maintenance
Enables local LLMs to search the web and fetch clean content from URLs without API keys, using SearxNG and Mozilla Readability.
Updated a year ago (2025-10-27 20:01 UTC)
2
40
MIT
mcp-internet
Browser Automation Web Scraping
Sonesh-2202
A
license
Not graded
quality
D
maintenance
Enables local LLMs to search the web, scrape pages, and extract structured data (tables, metadata) from sources like Wikipedia and IMDb, with caching and rate limiting.
Updated 17 days ago (2026-10-06 15:29 UTC)
1
MIT
browser-search-mcp
Browser Automation Web Scraping
fangsylar-pixel
A
license
A
quality
B
maintenance
Enables web search via a real browser for any LLM, bypassing anti-bot measures without API keys.
Updated 2 months ago (2026-08-12 16:20 UTC)
5
MIT
mcp-web
Browser Automation Web Scraping Search
MrDadpool
F
license
A
quality
C
maintenance
Enables a locally-run LLM to search the web, fetch pages as markdown, make arbitrary HTTP requests, and optionally render pages with headless Chromium.
Updated 2 months ago (2026-09-01 18:36 UTC)
3
-