Skip to main content
Glama
hlavacm

solocrawl

by hlavacm
OverviewSchemaRelated ServersScoreDiscussions
Python
Local

SoloCrawl is a small, self-hosted, fully async Python tool that does three things well: it searches the web across many sources at once, scrapes pages into clean markdown that's ready for an LLM, and looks up the latest package version from official registries. It runs on your machine, for free, and works the moment you install it — nothing to sign up for, no keys to paste.

It's built for individual developers and people tinkering with local LLMs. Use it as an MCP server (LM Studio, Claude Desktop, OpenCode, …), straight from the CLI, or as a Python library.

✨ Highlights

🔌 Zero config

Search, scrape, and package lookup all work out of the box — no accounts, no keys.

🔎 Federated search

Queries 11 possible sources, merges them with Reciprocal Rank Fusion and de-duplicates URLs into one clean ranking.

📄 Smart scraping

HTML → tidy markdown via trafilatura/readability, with a Playwright browser fallback for JS-heavy pages — used only when it's actually needed.

📦 Live package versions

10 ecosystems (PyPI, npm, crates.io, Maven, Go, …) resolved live from official registries — never a stale local DB.

🤖 MCP-native

Drops straight into local LLM tooling as a stdio MCP server with five ready tools.

⚡ Fully async, bounded

One shared HTTP client, one recycled browser, global + per-domain concurrency limits. Fast without hammering anyone.

🧩 Hackable

Add a search or package provider as a single self-registering file — the core stays untouched.

🔒 Safe by default

Blocks localhost/private/cloud-metadata targets and honours robots.txt.

Related MCP server: Local-MCP-server

🚀 Quick start

The recommended way to install SoloCrawl is pipx — it drops the solocrawl and solocrawl-mcp commands onto your PATH in their own isolated environment, so you can run them from anywhere without juggling a virtualenv:

git clone https://github.com/hlavacm/solocrawl.git
pipx install ./solocrawl     # or an absolute path: pipx install /path/to/solocrawl

Already on PyPI? Then it's just pipx install solocrawl — no checkout needed.

That's it — now run the three core commands from any directory:

# Scrape a page to markdown
solocrawl scrape https://example.com

# Federated web search (Wikipedia + DuckDuckGo + StackExchange by default)
solocrawl search "python asyncio semaphore" --limit 5

# Live package version lookup
solocrawl package requests --ecosystem pypi

Updating

# Installed from PyPI:
pipx upgrade solocrawl

# Installed from a local checkout — pull the latest changes, then reinstall:
cd /path/to/solocrawl && git pull && pipx install --force .   # alias: pipx reinstall solocrawl

After upgrading, restart your MCP client (LM Studio, Cursor, Claude Desktop, …) so it picks up the new solocrawl-mcp binary — your mcp.json needs no changes as long as it points at solocrawl-mcp on your PATH.

What you can do

🔎 Search the web

One query, many sources, a single merged ranking — no single provider deciding everything for you.

solocrawl search "python asyncio semaphore" --limit 5

# Pick exactly which sources to hit, and get machine-readable output
solocrawl search "django orm" --sources wikipedia,stackexchange --json

📄 Scrape a page to clean markdown

Turn any URL into LLM-ready markdown with page metadata (title, author, date, …) as front-matter.

solocrawl scrape https://example.com

# Save to a file, or force the browser for a JS-rendered page
solocrawl scrape https://example.com --out page.md
solocrawl scrape https://example.com --force-browser

📦 Look up package versions

The current version — and the one matching your constraint — straight from the official registry.

solocrawl package react --ecosystem npm --constraint ">=18,<19"
solocrawl package monolog/monolog --ecosystem packagist --json
solocrawl package some-lib --ecosystem pypi --allow-prerelease

🧪 Research in one shot

The classic LLM workflow — search, scrape the top hits, and get back one aggregated, cited report.

solocrawl research "python asyncio semaphore" --depth 3

🗂️ Batch-scrape many URLs

Fetch a whole list at once under the same bounded concurrency; --out-dir writes one file per URL.

solocrawl batch https://example.com https://www.python.org --out-dir /tmp/scrape
solocrawl batch --from-file urls.txt --out-dir /tmp/scrape

…and see what's available

# List every registered provider (search + package), default vs. opt-in
solocrawl providers

🤖 Use it with your local LLM (MCP)

This is where SoloCrawl really shines — give your local model (LM Studio, OpenCode, Claude Desktop, …) the ability to search, scrape, and check versions. The pipx install from the Quick start already put solocrawl-mcp on your PATH, so all that's left is pointing your MCP client at it.

LM Studio / Claude Desktop — ready-to-use config at examples/mcp.json. Drop it into your client's MCP settings (mcp.json):

{
  "mcpServers": {
    "solocrawl": {
      "command": "solocrawl-mcp",
      "args": [],
      "env": {
        "SOLOCRAWL_LOG_LEVEL": "INFO",
        "SOLOCRAWL_LOG_FILE": "~/.local/state/solocrawl/mcp.log"
      }
    }
  }
}

OpenCode — uses a different config format. Copy examples/opencode.jsonc into ~/.config/opencode/opencode.jsonc (global) or opencode.jsonc in your project root. OpenCode expects type: "local", command as an array, and environment instead of env.

If your MCP client doesn't inherit your shell PATH, replace "solocrawl-mcp" with the full path from which solocrawl-mcp (typically ~/.local/bin/solocrawl-mcp after pipx install). Logs go to stderr (visible in LM Studio Developer Logs) and optionally to SOLOCRAWL_LOG_FILE.

The server exposes five tools:

  • web_search(query, limit=5, sources=None) — federated search across enabled providers

  • scrape(url) — fetch and extract markdown (with page metadata) from a URL

  • research(query, depth=3) — search, scrape the top results, and return an aggregated cited report

  • package_version(name, ecosystem, constraint=None, allow_prerelease=False) — live registry lookup

  • list_providers(provider_type="all") — list registered search/package providers (default vs. opt-in)

To check the active version and command path:

pipx list | grep solocrawl
which solocrawl-mcp

Working from a local clone? A pipx-installed solocrawl-mcp is a snapshot — editing the repo does not update the command on your PATH, so your MCP client keeps running the old code. After changing the source, refresh it with pipx install --force . (or install once with pipx install --editable . so future edits are picked up automatically).

🐍 Use it from Python

import asyncio

from solocrawl.config import load_config
from solocrawl.core.search import federated_search, select_providers
from solocrawl.core.search.providers import duckduckgo, stackexchange, wikipedia  # noqa: F401

async def main() -> None:
    providers = select_providers(load_config())
    results = await federated_search(providers, "asyncio python", limit=3)
    for result in results:
        print(result.title, result.url)

asyncio.run(main())

See examples/library_search.py for a runnable example.

Search providers

Default (zero-config, always enabled):

Provider

Source

wikipedia

MediaWiki API

duckduckgo

ddgs package

stackexchange

Stack Exchange API (Stack Overflow)

Opt-in (enable with SOLOCRAWL_ENABLE_PROVIDERS):

Provider

Source

wikidata

Wikidata entity search

hackernews

Hacker News (Algolia)

arxiv

arXiv Atom API

pubmed

PubMed/NCBI E-utilities

github

GitHub repository search

mdn

MDN Web Docs search

reddit

Reddit post search (search.json)

searxng

Self-hosted SearXNG (set SOLOCRAWL_SEARXNG_URL)

SOLOCRAWL_ENABLE_PROVIDERS=arxiv,hackernews solocrawl search "transformer attention" --limit 6
SOLOCRAWL_ENABLE_PROVIDERS=github,mdn solocrawl search "fetch api" --limit 6

Package registries

Default ecosystems: PyPI, npm, Packagist, crates.io, NuGet, Maven Central, RubyGems, Go modules, pub.dev, Swift. Versions are always fetched live from official registries — SoloCrawl does not maintain its own version database. Swift packages have no central registry, so versions come from the repository's git tags (owner/repo on GitHub).

solocrawl package serde --ecosystem crates
solocrawl package Newtonsoft.Json --ecosystem nuget
solocrawl package org.junit.jupiter:junit-jupiter --ecosystem maven
solocrawl package github.com/gorilla/mux --ecosystem go
solocrawl package apple/swift-argument-parser --ecosystem swift

Optional extras

# Browser fallback for JS-heavy pages (Playwright)
pip install -e ".[browser]"
playwright install chromium

solocrawl scrape https://example.com --force-browser

# Install everything
pip install -e ".[all]"

⚙️ Configuration

All defaults work with no configuration. Everything below is optional and uses the SOLOCRAWL_ prefix. For local development, copy .env.dist to .env and uncomment what you need — SoloCrawl loads .env automatically via python-dotenv, and existing shell environment variables take precedence.

Variable

Default

Purpose

SOLOCRAWL_ENABLE_PROVIDERS

(empty)

Comma-separated opt-in provider names

SOLOCRAWL_SEARXNG_URL

(empty)

Base URL of a self-hosted SearXNG instance (enables the searxng provider)

SOLOCRAWL_RESPECT_ROBOTS

true

Honour robots.txt on scrape (fail-open); set false to skip

SOLOCRAWL_CACHE_TTL_SECONDS

0

In-memory fetch cache TTL in seconds (0 = disabled)

SOLOCRAWL_MAX_CONCURRENCY

10

Global fetch concurrency limit

SOLOCRAWL_PER_DOMAIN_LIMIT

2

Per-domain concurrency limit

SOLOCRAWL_TIMEOUT_SECONDS

30

Per-request timeout in seconds

SOLOCRAWL_MAX_RETRIES

3

Retries on network errors / rate limits

SOLOCRAWL_MAX_RESPONSE_BYTES

10485760

Cap on fetched response body size (10 MiB); larger bodies are truncated

SOLOCRAWL_PROXY_ENABLED

false

Enable optional proxy layer

SOLOCRAWL_PROXY_MODE

list

Proxy mode: list (rotate a pool) or endpoint (single rotating endpoint)

SOLOCRAWL_PROXY_LIST

(empty)

Comma-separated proxy URLs

SOLOCRAWL_PROXY_ENDPOINT

(empty)

Single rotating proxy endpoint

SOLOCRAWL_PROXY_USERNAME

(empty)

Proxy auth username

SOLOCRAWL_PROXY_PASSWORD

(empty)

Proxy auth password

SOLOCRAWL_ALLOW_INTERNAL_URLS

false

Allow scraping localhost/private IPs (dev only)

SOLOCRAWL_USER_AGENT

(SoloCrawl default)

Override HTTP User-Agent for API requests

SOLOCRAWL_BROWSER_ALLOWED

true

Allow Playwright fallback when installed

SOLOCRAWL_LOG_LEVEL

WARNING

Log level: DEBUG, INFO, WARNING, ERROR

SOLOCRAWL_LOG_FILE

(empty)

Optional log file path (also logs to stderr)

🔒 Security note on URL fetching

By default SoloCrawl refuses to fetch localhost, link-local, private, reserved, and cloud-metadata addresses. It checks literal hosts, DNS-resolved A/AAAA records, HTTP redirect targets, and Playwright's final browser URL. SoloCrawl is still a single-user local tool, not a hostile-multi-tenant proxy — do not expose it to untrusted network callers. SOLOCRAWL_ALLOW_INTERNAL_URLS=true disables these internal-target checks entirely (intended for trusted local development only).

🧩 Extending it

The whole point of the plugin layout is that adding a source is a single self-registering file — the core never changes. To add a search provider:

  1. Create src/solocrawl/core/search/providers/myprovider.py implementing SearchProvider.

  2. Register with @register("myprovider", zero_config=True) or as opt-in.

  3. Import the module in src/solocrawl/core/search/providers/__init__.py so registration runs.

  4. Add fixture-based tests in tests/.

The same pattern applies to package providers under src/solocrawl/core/packages/providers/.

Development

Work from a checkout in a virtualenv with an editable install — this also drops the solocrawl and solocrawl-mcp scripts into .venv/bin/:

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

Then run the quality gate:

ruff check . && ruff format --check .
pyright
pytest

# …or all in one line:
ruff check . && pyright && pytest

Ethics and terms of use

SoloCrawl is built for individual developers and local LLM tooling. It respects the robots.txt and terms of service of target sites — scrape consults robots.txt and refuses disallowed URLs by default (fail-open on errors; opt out with SOLOCRAWL_RESPECT_ROBOTS=false). The proxy and scraping features are not intended to bypass site rules, captchas, or anti-bot systems. Use responsibly and stay within legitimate access patterns.

License

MIT — see LICENSE.

Available Tools

5 tools
list_providersA

List the registered search and package providers (default vs. opt-in).

ParametersJSON Schema
NameRequiredDescriptionDefault
provider_typeNoWhich providers to list: 'search', 'package', or 'all'.all

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Without annotations, the description carries the burden of behavioral disclosure. It adds context about default vs. opt-in providers, but does not mention safety, authentication, or side effects, which is a moderate gap for a read-only listing tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence of 12 words, front-loaded with the action and resource, with no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has one simple optional parameter and an output schema, the description covers the necessary context—listing both search and package providers with default vs. opt-in distinction—without needing to explain return values due to the output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds extra meaning by explaining that providers are listed as default vs. opt-in, which is not in the schema parameter description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'List' and the resource 'registered search and package providers', and distinguishes between default vs. opt-in, making it distinct from sibling tools like web_search or package_version.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use this tool (to list providers), but does not provide explicit guidance on when not to use it or mention alternatives, though the purpose is clear enough in context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

package_versionB

Look up the latest or constraint-satisfying version of a package from a registry.

ParametersJSON Schema
NameRequiredDescriptionDefault
nameYesPackage name (Packagist: vendor/package, Maven: groupId:artifactId, Go: module path, Swift: owner/repo on GitHub).
ecosystemYesRegistry ecosystem (pypi, npm, packagist, crates, nuget, maven, rubygems, go, pub, swift).
constraintNoOptional version constraint such as '>=4.2,<5'.
allow_prereleaseNoWhether pre-release versions may be selected.

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It only states the basic operation and does not explain side effects, error handling, caching behavior, or constraints like network dependency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence that immediately conveys the tool's core function. It is front-loaded with the key verb and resource, with no extraneous information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given that an output schema exists (not shown but indicated) and parameters are fully described, the description is adequate but minimal. It lacks details on behavior for edge cases like non-existent packages or constraint satisfaction.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description does not add significant meaning beyond the schema; it repeats the purpose but does not elaborate on parameter usage or relationships.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's verb ('look up') and resource ('latest or constraint-satisfying version of a package'), and the specific action distinguishes it from sibling tools like web_search or scrape which are more general.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives, nor does it mention exclusion criteria or prerequisites. The description leaves the agent to infer usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

researchA

Search the web, scrape the top results, and return an aggregated cited report.

ParametersJSON Schema
NameRequiredDescriptionDefault
depthNoHow many top results to scrape and aggregate (default 3).
queryYesThe research query.

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Without annotations, the description discloses the multi-step process (search, scrape, aggregate, cite), which is fairly transparent. However, specifics like number of results searched or citation format are omitted.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, clear sentence without superfluous words, front-loaded with the key action 'Search the web'.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity and presence of an output schema, the description covers the essential workflow and output format, though it could mention limitations like the number of initial search results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with clear descriptions for query and depth. The tool's description adds no additional parameter meaning beyond the schema, so baseline score applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool searches the web, scrapes top results, and returns an aggregated cited report, distinguishing it from sibling tools like web_search (search only) and scrape (scrape only).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied (for research tasks requiring aggregation), but there is no explicit guidance on when to use this tool versus alternatives like web_search or scrape.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrapeA

Fetch a URL and return the main page content as markdown suitable for LLM context.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe HTTP or HTTPS URL to scrape.

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description must fully convey behavioral traits. It states the action (fetch, return markdown) but omits potential side effects, rate limits, authentication needs, or limitations (e.g., no JS rendering). Adequate but not rich.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence with no extraneous information. Every word serves the purpose: verb, resource, output, and context. Efficient and front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (one required parameter) and existence of an output schema, the description adequately covers what the agent needs. Return format is specified, and schema handles remaining details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% (url parameter fully described in schema). Description adds context about output format but does not add new meaning to the parameter itself. Baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description uses specific verb 'Fetch', identifies resource 'URL', specifies output format 'markdown', and clarifies suitability 'for LLM context'. Clearly distinguishes from sibling tools like web_search (search results) and research (deeper investigation).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Description implies use when a specific URL's content is needed, but provides no explicit guidance on when not to use or alternatives. Sibling tools exist (web_search, research) but no mention of their differentiation.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A3.6/5.0
Disambiguation3/5

Tools like research and web_search both perform web searches, with research also scraping and aggregating results. Similarly, scrape is available alongside research's scraping capability. While descriptions differentiate them, the boundaries could confuse an agent.

Naming Consistency2/5

Tool names mix patterns: list_providers follows verb_noun, package_version uses noun_noun, while research and scrape are bare verbs, and web_search is noun_noun. No consistent style is maintained.

Tool Count5/5

Five tools cover the core functionality—provider listing, package version lookup, web search, scraping, and combined research—without being excessive or insufficient for the server's stated purpose.

Completeness4/5

The tool set covers primary use cases (search, scrape, package lookup) but lacks features like crawling multiple pages or provider management, which are minor gaps for a research-focused server.

Maintenance

ActivitySlowing
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • WebFetch.MCP

    manooll
    A
    license
    A
    quality
    C
    maintenance
    Enables local LLMs to search the web and fetch clean content from URLs without API keys, using SearxNG and Mozilla Readability.
    2
    36
    MIT

  • mcp-internet

    Sonesh-2202
    A
    license
    Not graded
    quality
    C
    maintenance
    Enables local LLMs to search the web, scrape pages, and extract structured data (tables, metadata) from sources like Wikipedia and IMDb, with caching and rate limiting.
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/hlavacm/solocrawl'

If you have feedback or need assistance with the MCP directory API, please join our Discord server