solocrawl

SoloCrawl is a small, self-hosted, fully async Python tool that does three things well: it searches the web across many sources at once, scrapes pages into clean markdown that's ready for an LLM, and looks up the latest package version from official registries. It runs on your machine, for free, and works the moment you install it — nothing to sign up for, no keys to paste.

It's built for individual developers and people tinkering with local LLMs. Use it as an MCP server (LM Studio, Claude Desktop, OpenCode, …), straight from the CLI, or as a Python library.

✨ Highlights

Related MCP server: Local-MCP-server

🚀 Quick start

The recommended way to install SoloCrawl is pipx — it drops the solocrawl and solocrawl-mcp commands onto your PATH in their own isolated environment, so you can run them from anywhere without juggling a virtualenv:

git clone https://github.com/hlavacm/solocrawl.git
pipx install ./solocrawl     # or an absolute path: pipx install /path/to/solocrawl

Already on PyPI? Then it's just pipx install solocrawl — no checkout needed.

That's it — now run the three core commands from any directory:

# Scrape a page to markdown
solocrawl scrape https://example.com

# Federated web search (Wikipedia + DuckDuckGo + StackExchange by default)
solocrawl search "python asyncio semaphore" --limit 5

# Live package version lookup
solocrawl package requests --ecosystem pypi

Updating

# Installed from PyPI:
pipx upgrade solocrawl

# Installed from a local checkout — pull the latest changes, then reinstall:
cd /path/to/solocrawl && git pull && pipx install --force .   # alias: pipx reinstall solocrawl

After upgrading, restart your MCP client (LM Studio, Cursor, Claude Desktop, …) so it picks up the new solocrawl-mcp binary — your mcp.json needs no changes as long as it points at solocrawl-mcp on your PATH.

What you can do

🔎 Search the web

One query, many sources, a single merged ranking — no single provider deciding everything for you.

solocrawl search "python asyncio semaphore" --limit 5

# Pick exactly which sources to hit, and get machine-readable output
solocrawl search "django orm" --sources wikipedia,stackexchange --json

📄 Scrape a page to clean markdown

Turn any URL into LLM-ready markdown with page metadata (title, author, date, …) as front-matter.

solocrawl scrape https://example.com

# Save to a file, or force the browser for a JS-rendered page
solocrawl scrape https://example.com --out page.md
solocrawl scrape https://example.com --force-browser

📦 Look up package versions

The current version — and the one matching your constraint — straight from the official registry.

solocrawl package react --ecosystem npm --constraint ">=18,<19"
solocrawl package monolog/monolog --ecosystem packagist --json
solocrawl package some-lib --ecosystem pypi --allow-prerelease

🧪 Research in one shot

The classic LLM workflow — search, scrape the top hits, and get back one aggregated, cited report.

solocrawl research "python asyncio semaphore" --depth 3

🗂️ Batch-scrape many URLs

Fetch a whole list at once under the same bounded concurrency; --out-dir writes one file per URL.

solocrawl batch https://example.com https://www.python.org --out-dir /tmp/scrape
solocrawl batch --from-file urls.txt --out-dir /tmp/scrape

…and see what's available

# List every registered provider (search + package), default vs. opt-in
solocrawl providers

🤖 Use it with your local LLM (MCP)

This is where SoloCrawl really shines — give your local model (LM Studio, OpenCode, Claude Desktop, …) the ability to search, scrape, and check versions. The pipx install from the Quick start already put solocrawl-mcp on your PATH, so all that's left is pointing your MCP client at it.

LM Studio / Claude Desktop — ready-to-use config at examples/mcp.json. Drop it into your client's MCP settings (mcp.json):

{
  "mcpServers": {
    "solocrawl": {
      "command": "solocrawl-mcp",
      "args": [],
      "env": {
        "SOLOCRAWL_LOG_LEVEL": "INFO",
        "SOLOCRAWL_LOG_FILE": "~/.local/state/solocrawl/mcp.log"
      }
    }
  }
}

OpenCode — uses a different config format. Copy examples/opencode.jsonc into ~/.config/opencode/opencode.jsonc (global) or opencode.jsonc in your project root. OpenCode expects type: "local", command as an array, and environment instead of env.

If your MCP client doesn't inherit your shell PATH, replace "solocrawl-mcp" with the full path from which solocrawl-mcp (typically ~/.local/bin/solocrawl-mcp after pipx install). Logs go to stderr (visible in LM Studio Developer Logs) and optionally to SOLOCRAWL_LOG_FILE.

The server exposes five tools:

• web_search(query, limit=5, sources=None) — federated search across enabled providers

• scrape(url) — fetch and extract markdown (with page metadata) from a URL

• research(query, depth=3) — search, scrape the top results, and return an aggregated cited report

• package_version(name, ecosystem, constraint=None, allow_prerelease=False) — live registry lookup

• list_providers(provider_type="all") — list registered search/package providers (default vs. opt-in)

To check the active version and command path:

pipx list | grep solocrawl
which solocrawl-mcp

Working from a local clone? A pipx-installed solocrawl-mcp is a snapshot — editing the repo does not update the command on your PATH, so your MCP client keeps running the old code. After changing the source, refresh it with pipx install --force . (or install once with pipx install --editable . so future edits are picked up automatically).

🐍 Use it from Python

import asyncio

from solocrawl.config import load_config
from solocrawl.core.search import federated_search, select_providers
from solocrawl.core.search.providers import duckduckgo, stackexchange, wikipedia  # noqa: F401

async def main() -> None:
    providers = select_providers(load_config())
    results = await federated_search(providers, "asyncio python", limit=3)
    for result in results:
        print(result.title, result.url)

asyncio.run(main())

See examples/library_search.py for a runnable example.

Search providers

Default (zero-config, always enabled):

Opt-in (enable with SOLOCRAWL_ENABLE_PROVIDERS):

SOLOCRAWL_ENABLE_PROVIDERS=arxiv,hackernews solocrawl search "transformer attention" --limit 6
SOLOCRAWL_ENABLE_PROVIDERS=github,mdn solocrawl search "fetch api" --limit 6

Package registries

Default ecosystems: PyPI, npm, Packagist, crates.io, NuGet, Maven Central, RubyGems, Go modules, pub.dev, Swift. Versions are always fetched live from official registries — SoloCrawl does not maintain its own version database. Swift packages have no central registry, so versions come from the repository's git tags (owner/repo on GitHub).

solocrawl package serde --ecosystem crates
solocrawl package Newtonsoft.Json --ecosystem nuget
solocrawl package org.junit.jupiter:junit-jupiter --ecosystem maven
solocrawl package github.com/gorilla/mux --ecosystem go
solocrawl package apple/swift-argument-parser --ecosystem swift

Optional extras

# Browser fallback for JS-heavy pages (Playwright)
pip install -e ".[browser]"
playwright install chromium

solocrawl scrape https://example.com --force-browser

# Install everything
pip install -e ".[all]"

⚙️ Configuration

All defaults work with no configuration. Everything below is optional and uses the SOLOCRAWL_ prefix. For local development, copy .env.dist to .env and uncomment what you need — SoloCrawl loads .env automatically via python-dotenv, and existing shell environment variables take precedence.

🔒 Security note on URL fetching

By default SoloCrawl refuses to fetch localhost, link-local, private, reserved, and cloud-metadata addresses. It checks literal hosts, DNS-resolved A/AAAA records, HTTP redirect targets, and Playwright's final browser URL. SoloCrawl is still a single-user local tool, not a hostile-multi-tenant proxy — do not expose it to untrusted network callers. SOLOCRAWL_ALLOW_INTERNAL_URLS=true disables these internal-target checks entirely (intended for trusted local development only).

🧩 Extending it

The whole point of the plugin layout is that adding a source is a single self-registering file — the core never changes. To add a search provider:

1. Create src/solocrawl/core/search/providers/myprovider.py implementing SearchProvider.

2. Register with @register("myprovider", zero_config=True) or as opt-in.

3. Import the module in src/solocrawl/core/search/providers/__init__.py so registration runs.

4. Add fixture-based tests in tests/.

The same pattern applies to package providers under src/solocrawl/core/packages/providers/.

Development

Work from a checkout in a virtualenv with an editable install — this also drops the solocrawl and solocrawl-mcp scripts into .venv/bin/:

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

Then run the quality gate:

ruff check . && ruff format --check .
pyright
pytest

# …or all in one line:
ruff check . && pyright && pytest

Ethics and terms of use

SoloCrawl is built for individual developers and local LLM tooling. It respects the robots.txt and terms of service of target sites — scrape consults robots.txt and refuses disallowed URLs by default (fail-open on errors; opt out with SOLOCRAWL_RESPECT_ROBOTS=false). The proxy and scraping features are not intended to bypass site rules, captchas, or anti-bot systems. Use responsibly and stay within legitimate access patterns.

License

MIT — see LICENSE.