opencrawl
by ktcod
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| HOST | No | HTTP API bind host | 127.0.0.1 |
| PORT | No | HTTP API bind port | 3002 |
| SEARXNG_URL | No | SearXNG server URL; enables web, news and images sources | http://localhost:8080 |
| OPENCRAWL_DB | No | SQLite store path (`:memory:` for a throwaway store) | ~/.opencrawl/opencrawl.db |
| OPENCRAWL_API_KEY | No | Bearer token required by the HTTP API. Required to bind a non-loopback `HOST` (or set `OPENCRAWL_INSECURE=1`) | |
| OPENCRAWL_BROWSER | No | `auto` tries bundled Chromium, then Chrome, then Edge. `chromium`, `chrome`, `msedge`, or `none` | auto |
| OPENCRAWL_RETRIES | No | Retries on HTTP 429/503 per fetch | 2 |
| OPENCRAWL_DATA_DIR | No | Moves the opencrawl data folder | |
| OPENCRAWL_MAP_DEPTH | No | How far map follows links when the sitemap is thin | 1 |
| OPENCRAWL_MAP_LIMIT | No | Default map URL cap | 5000 |
| OPENCRAWL_ROBOTS_UA | No | Token checked against robots.txt during crawls | opencrawl |
| BRAVE_SEARCH_API_KEY | No | Brave Search API key; enables web, news and images sources | |
| OPENCRAWL_TIMEOUT_MS | No | Per-request timeout | 30000 |
| OPENCRAWL_USER_AGENT | No | User agent for HTTP and browser requests | Chrome desktop UA |
| OPENCRAWL_CRAWL_LIMIT | No | Default crawl page cap | 100 |
| OPENCRAWL_TOOL_PREFIX | No | MCP tool name prefix | |
| OPENCRAWL_ALLOWED_HOSTS | No | Extra `Host` header values the HTTP API accepts (DNS-rebinding guard); `*` disables the check | bind host |
| OPENCRAWL_MAP_MAX_PAGES | No | How far map follows links when the sitemap is thin | 20 |
| OPENCRAWL_SEARCH_BACKEND | No | Force one search backend: `duckduckgo|searxng|brave` | |
| OPENCRAWL_ALLOWED_ORIGINS | No | Browser origins allowed to call the HTTP API; any other `Origin` header is refused | |
| OPENCRAWL_CRAWL_PAGE_SIZE | No | Documents per page on `GET /v2/crawl/:id` | 200 |
| OPENCRAWL_BATCH_CONCURRENCY | No | Parallel pages for batch scrapes | 5 |
| OPENCRAWL_CRAWL_CONCURRENCY | No | Parallel pages per crawl (max 20) | 5 |
| OPENCRAWL_MAX_RETRY_WAIT_MS | No | Longest wait between retries, even if Retry-After asks for more | 30000 |
| OPENCRAWL_DEFAULT_MAX_AGE_MS | No | Cache window used when a request sets no `maxAge` | 0 |
| OPENCRAWL_JOB_RETENTION_DAYS | No | How long finished crawl jobs stay in the store | 7 |
| OPENCRAWL_MAX_RESPONSE_BYTES | No | Largest page or sitemap body buffered (50 MB) | 52428800 |
| OPENCRAWL_MAX_SITEMAP_FETCHES | No | Sitemap files fetched per map or crawl | 50 |
| OPENCRAWL_MAX_BROWSER_CONTEXTS | No | Browser pages open at once across all scrapes | 4 |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Server capabilities have not been inspected yet.
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
No tools | |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo issues