# crawlport

crawlport is a doorway between an agent and the readable web, backed
by Firecrawl. Give it one page and it hands back clean markdown; give
it a whole site and it will map the pages, crawl a section, or scrape
a list in batch; give it a question and it can search, pull structured
fields out of pages to a schema you describe, research across many
sources, or draft the llms.txt a site would publish about itself. It
runs as a CLI or as an MCP server, and it needs a Firecrawl API key.

## Setup

Get a key from Firecrawl and export it:

```bash
export FIRECRAWL_API_KEY=<redacted>
```

Without the variable set, every command and tool call fails with an
error that names `FIRECRAWL_API_KEY` — nothing is sent anywhere.

## Install

```bash
npx @reneturcios/crawlport scrape https://example.com/some/page
```

Or run the container image:

```bash
docker run -i -e FIRECRAWL_API_KEY ghcr.io/cloudcorprecords/crawlport
```

The container defaults to the MCP server (see below). Pass CLI
arguments to use it as a CLI instead.

## CLI

```bash
# One page, as markdown (add --formats html,links etc. for more)
crawlport scrape https://example.com/article --only-main-content

# Discover a site's URLs, optionally filtered
crawlport map https://example.com --search pricing --limit 100

# Web search, with locale controls
crawlport search "standing desk reviews" --limit 5 --lang en --country US

# Start a crawl, then poll it with the returned job id
crawlport crawl https://example.com/docs --limit 50 --max-depth 2
crawlport crawl-status <job-id>

# Structured extraction and batch scraping
crawlport extract https://example.com/a https://example.com/b --prompt "pull the price"
crawlport batch-scrape https://example.com/a https://example.com/b
crawlport batch-status <job-id>

# Research and llms.txt
crawlport deep-research "home battery storage costs in 2026" --max-urls 40
crawlport llmstxt https://example.com --max-urls 25

crawlport --version
```

Successful calls print the API's JSON response. Crawl and batch jobs
are asynchronous upstream: the start call returns a job id, and the
matching status command reads its progress and results. Exit codes:
`0` on success, `1` when the API call fails (including a missing
key), `2` for usage errors.

## MCP server

One bin serves both faces. The `mcp` subcommand starts a stdio MCP
server:

```bash
npx -y @reneturcios/crawlport mcp
```

MCP client configuration:

```json
{
  "mcpServers": {
    "crawlport": {
      "command": "npx",
      "args": ["-y", "@reneturcios/crawlport", "mcp"],
      "env": { "FIRECRAWL_API_KEY": "your-key-here" }
    }
  }
}
```

### Tools

| Tool | Input | What it does |
| --- | --- | --- |
| `firecrawl_scrape` | `url` (required); `formats`, `onlyMainContent`, `waitFor`, `timeout`, `actions`, `includeTags`, `excludeTags`, `blockAds`, `proxy` (optional) | Scrape one page via `POST /v1/scrape`. |
| `firecrawl_map` | `url` (required); `search`, `limit`, `ignoreSitemap` | List a site's discovered URLs via `POST /v1/map`. |
| `firecrawl_search` | `query` (required); `limit`, `lang`, `country`, `scrapeOptions` | Web search via `POST /v1/search`, optionally scraping each result. |
| `firecrawl_crawl` | `url` (required); `limit`, `maxDepth`, `allowBackwardLinks`, `allowExternalLinks`, `excludePaths`, `includePaths`, `scrapeOptions` | Start a crawl via `POST /v1/crawl`; returns a job id. |
| `firecrawl_check_crawl_status` | `id` (required) | Poll a crawl job via `GET /v1/crawl/{id}`. |
| `firecrawl_extract` | `urls` (required); `prompt`, `schema`, `enableWebSearch`, `allowExternalLinks` | LLM extraction to a prompt/schema via `POST /v1/extract`. |
| `firecrawl_batch_scrape` | `urls` (required); `formats`, `onlyMainContent` | Start a batch scrape via `POST /v1/batch/scrape`; returns a job id. |
| `firecrawl_check_batch_status` | `id` (required) | Poll a batch job via `GET /v1/batch/scrape/{id}`. |
| `firecrawl_deep_research` | `query` (required); `maxDepth`, `timeLimit`, `maxUrls` | Multi-source research via `POST /v1/deep-research`. |
| `firecrawl_generate_llmstxt` | `url` (required); `maxUrls`, `showFullText` | Generate llms.txt content via `POST /v1/llmstxt`. |

Tool results are the upstream JSON, pretty-printed as text. Optional
arguments are forwarded only when provided, so the API's defaults
apply otherwise. Upstream failures come back as tool errors carrying
the HTTP status and the API's own message.

## Development

```bash
npm install
npm run build   # tsc -> dist/
npm test        # node test runner over test/*.test.ts, no network (fake fetch fixtures)
npm run dev     # run the CLI from source via tsx
```

## License

MIT — see `LICENSE`.