Web Research Stack

A self-hosted web research platform that exposes one stable interface for web search, discovery, scraping, crawling, structured extraction, and browser interaction.

The stack deliberately combines several specialized services rather than trying to make one component do everything:

Important: this Compose project does not start Ollama or SearXNG. It is designed to reuse the Ollama and SearXNG instances you already run separately.

Architecture

                          Applications / LangGraph / Agents
                                      |
                      +---------------+---------------+
                      |                               |
                 REST :8080                       MCP :8081
                      |                               |
                      +---------------+---------------+
                                      |
                                Gateway API
                                      |
             +------------------------+------------------------+
             |                         |                        |
       Firecrawl API              Extract API             Interact API
   search/map/scrape/crawl       structured JSON          Playwright
             |                         |                        |
             |                         +------ Ollama ----------+
             |
          SearXNG

External existing services:
  Ollama  -> http://host.docker.internal:11434
  SearXNG -> http://host.docker.internal:8088

Firecrawl also runs its own internal Playwright service for page scraping. The interact service is separate and exists specifically for stateful browser automation such as navigating, clicking, typing, selecting, scrolling, reading page text, and taking screenshots.

Related MCP server: Sentinel Core Agent

Design principles

• One stable contract: application code talks to the gateway rather than directly to Firecrawl, Ollama, or Playwright.

• REST and MCP use the same backend logic: the MCP server calls the REST gateway instead of duplicating implementation code.

• External Ollama and SearXNG: existing services are reused instead of creating duplicate containers.

• Deterministic first: browser interaction uses explicit selectors and accessible text before asking an LLM to resolve ambiguity.

• Structured extraction: Ollama output is validated against the caller's JSON Schema and retried when invalid.

• Evidence preservation: extract responses include source/provenance metadata.

• Browser isolation: each interact session receives its own Playwright browser context with a TTL.

• SSRF controls: browser navigation and subrequests reject private/internal destinations by default.

• Conservative automation: potentially consequential clicks are blocked unless explicitly allowed.

• Private dependencies: Redis, RabbitMQ, PostgreSQL, Firecrawl workers, and both Playwright services are not published to the host.

Project layout

web-research-stack/
├── docker-compose.yaml
├── .env.example
├── .dockerignore
├── .gitignore
├── Dockerfile.python
├── requirements.txt
├── Makefile
├── pytest.ini
├── common/
│   ├── http.py
│   └── logging.py
├── gateway/
│   └── app.py
├── extract/
│   └── app.py
├── interact/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── app.py
│   └── security.py
├── mcp/
│   └── server.py
├── scripts/
│   ├── lib.sh
│   ├── test-all.sh
│   ├── test-functionality.sh
│   ├── test-mcp.sh
│   └── test-security.sh
├── tests/
│   ├── live_mcp.py
│   ├── test_http_security.py
│   └── test_security.py
└── .github/workflows/
    └── ci.yml

1. Prerequisites

You need:

• Docker Engine

• Docker Compose v2

• an existing Ollama service

• an existing SearXNG service with JSON output enabled

• at least one Ollama model suitable for structured JSON, such as qwen2.5:14b

This project assumes your existing services are reachable from Docker containers through host-published ports:

Ollama:  http://host.docker.internal:11434
SearXNG: http://host.docker.internal:8088

On Linux, the Compose services that need these addresses include:

extra_hosts:
  - "host.docker.internal:host-gateway"

That lets the Web Research Stack use separate containers without requiring them to belong to the same Compose project or Docker network.

Verify the existing services first

From the Docker host:

curl -sS http://127.0.0.1:11434/api/tags | jq

For SearXNG:

curl -sS 'http://127.0.0.1:8088/search?q=firecrawl&format=json' | jq '.results[:2]'

If either command fails, fix that service before starting this project.

2. Configuration

Create the environment file:

cp .env.example .env

At minimum review these settings:

GATEWAY_API_KEY=replace-with-a-long-random-value
POSTGRES_PASSWORD=replace-this-postgres-password

OLLAMA_BASE_URL=http://host.docker.internal:11434
OLLAMA_MODEL=qwen2.5:14b

SEARXNG_ENDPOINT=http://host.docker.internal:8088
SEARXNG_ENGINES=
SEARXNG_CATEGORIES=general

SEARXNG_ENDPOINT is passed to Firecrawl, so /v1/search uses the SearXNG instance you already operate.

OLLAMA_BASE_URL is used directly by the dedicated extract service and by interact only when selector ambiguity requires LLM assistance.

The project does not create an ollama service or a searxng service in docker-compose.yaml.

If Ollama or SearXNG use different ports

Simply change the environment values:

OLLAMA_BASE_URL=http://host.docker.internal:11435
SEARXNG_ENDPOINT=http://host.docker.internal:8888

If you prefer container DNS names

If your existing services share an external Docker network with this project, you can instead use addresses such as:

OLLAMA_BASE_URL=http://ollama:11434
SEARXNG_ENDPOINT=http://searxng:8080

You must then attach the relevant Web Research Stack containers to that external Docker network. The supplied configuration uses host-published ports because it keeps the projects independent and requires no shared-network naming convention.

3. Firecrawl versioning

The example .env allows:

FIRECRAWL_VERSION=latest

That is convenient for an initial test, but production deployments should use an exact Firecrawl release or image digest you have tested.

Firecrawl's own self-hosting guidance recommends keeping the Compose configuration aligned with the release you deploy. Its internal service topology and environment variables can change across releases.

4. Start the stack

Validate the Compose file first:

docker compose config

Build and start:

docker compose up -d --build

or:

make up

Inspect status:

docker compose ps

Follow logs:

docker compose logs -f --tail=200

The two public interfaces are bound to loopback by default:

REST API:    http://127.0.0.1:8080
OpenAPI UI:  http://127.0.0.1:8080/docs
MCP server:  http://127.0.0.1:8081/mcp

5. REST API usage

The REST gateway is the preferred interface for conventional applications, Python services, LangGraph HTTP nodes, shell scripts, and integrations that already use HTTP APIs.

Set the gateway key once for shell examples:

export WRS_KEY='replace-with-your-GATEWAY_API_KEY'
export WRS='http://127.0.0.1:8080'

All gateway requests use:

Authorization: Bearer <GATEWAY_API_KEY>

REST endpoint summary

Search

curl -sS "$WRS/v1/search" \
  -H "Authorization: Bearer $WRS_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "accessible hotels Vancouver BC",
    "limit": 5
  }' | jq

Flow:

REST client -> Gateway -> Firecrawl /v2/search -> existing SearXNG

Map a website

curl -sS "$WRS/v1/map" \
  -H "Authorization: Bearer $WRS_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "limit": 100
  }' | jq

Scrape a page

curl -sS "$WRS/v1/scrape" \
  -H "Authorization: Bearer $WRS_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"]
  }' | jq

Start a crawl

curl -sS "$WRS/v1/crawl" \
  -H "Authorization: Bearer $WRS_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "limit": 50
  }' | jq

Save the returned crawl job ID and poll it:

JOB_ID='<returned-job-id>'

curl -sS "$WRS/v1/crawl/$JOB_ID" \
  -H "Authorization: Bearer $WRS_KEY" | jq

Extract structured data from a URL

The gateway first scrapes the page through Firecrawl and then sends the resulting content to the dedicated Ollama extraction service.

URL
 |
 v
Firecrawl scrape
 |
 v
markdown
 |
 v
Extract service
 |
 v
existing Ollama endpoint
 |
 v
JSON Schema validation + bounded retry

Example:

curl -sS "$WRS/v1/extract" \
  -H "Authorization: Bearer $WRS_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "instruction": "Extract the page title and a short summary using only the supplied page content.",
    "schema": {
      "type": "object",
      "properties": {
        "title": {"type": ["string", "null"]},
        "summary": {"type": ["string", "null"]}
      },
      "required": ["title", "summary"],
      "additionalProperties": false
    }
  }' | jq

Typical response shape:

{
  "data": {
    "title": "Example Domain",
    "summary": "..."
  },
  "provenance": {
    "source_url": "https://example.com",
    "content_sha256": "...",
    "content_chars": 1234,
    "truncated": false,
    "model": "qwen2.5:14b",
    "attempts": 1,
    "generated_at": "..."
  }
}

Extract from content you already have

This avoids an unnecessary scrape:

curl -sS "$WRS/v1/extract" \
  -H "Authorization: Bearer $WRS_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "content": "Acme Inc. is headquartered in Vancouver, British Columbia.",
    "instruction": "Extract company and headquarters.",
    "schema": {
      "type": "object",
      "properties": {
        "company": {"type": ["string", "null"]},
        "headquarters": {"type": ["string", "null"]}
      },
      "required": ["company", "headquarters"],
      "additionalProperties": false
    }
  }' | jq

Browser interaction

Create a session

SESSION_ID=$(curl -sS "$WRS/v1/interact/sessions" \
  -X POST \
  -H "Authorization: Bearer $WRS_KEY" \
  -H 'Content-Type: application/json' \
  -d '{}' | jq -r '.session_id')

echo "$SESSION_ID"

Navigate

curl -sS "$WRS/v1/interact/sessions/$SESSION_ID/navigate" \
  -H "Authorization: Bearer $WRS_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com"}' | jq

Click with an explicit selector

curl -sS "$WRS/v1/interact/sessions/$SESSION_ID/action" \
  -H "Authorization: Bearer $WRS_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "action": "click",
    "selector": "a"
  }' | jq

Type into a field

curl -sS "$WRS/v1/interact/sessions/$SESSION_ID/action" \
  -H "Authorization: Bearer $WRS_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "action": "type",
    "selector": "input[name=q]",
    "value": "British Columbia"
  }' | jq

Use a natural-language element description

When no CSS selector is supplied, the service first tries deterministic accessible-text and label matching. Ollama is used only when the target remains ambiguous.

curl -sS "$WRS/v1/interact/sessions/$SESSION_ID/action" \
  -H "Authorization: Bearer $WRS_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "action": "click",
    "description": "Search"
  }' | jq

Read page text

curl -sS "$WRS/v1/interact/sessions/$SESSION_ID/text" \
  -H "Authorization: Bearer $WRS_KEY" | jq

Save a screenshot

curl -sS "$WRS/v1/interact/sessions/$SESSION_ID/screenshot" \
  -H "Authorization: Bearer $WRS_KEY" \
  -o screenshot.png

Close the session

curl -sS "$WRS/v1/interact/sessions/$SESSION_ID" \
  -X DELETE \
  -H "Authorization: Bearer $WRS_KEY" | jq

6. MCP usage

The MCP server is an alternate interface over the same REST gateway. It does not maintain a second implementation of search, scrape, crawl, extract, or browser interaction.

MCP client
   |
   v
http://127.0.0.1:8081/mcp
   |
   v
FastMCP server
   |
   v
REST gateway
   |
   +--> Firecrawl --> SearXNG
   +--> Extract --> Ollama
   +--> Interact --> Playwright --> optional Ollama

The MCP endpoint uses Streamable HTTP:

http://127.0.0.1:8081/mcp

Because the Compose file binds MCP to 127.0.0.1, it is accessible only from the Docker host by default. This is intentional. If you expose MCP to another machine, put it behind TLS and authentication rather than simply changing the bind address.

MCP tools

Use MCP from a FastMCP Python client

Install the client library in your calling environment:

pip install fastmcp

Example:

import asyncio
from fastmcp import Client


async def main():
    async with Client("http://127.0.0.1:8081/mcp") as client:
        tools = await client.list_tools()
        print([tool.name for tool in tools])

        result = await client.call_tool(
            "search",
            {
                "query": "accessible hotels Vancouver BC",
                "limit": 5,
            },
        )
        print(result)


asyncio.run(main())

MCP search example

Call tool:

search

Arguments:

{
  "query": "Royal Caribbean Alaska 2027",
  "limit": 10
}

This executes the same backend path as:

POST /v1/search

MCP scrape example

Tool:

scrape

Arguments:

{
  "url": "https://example.com",
  "formats": ["markdown"]
}

MCP extract example

Tool:

extract

Arguments:

{
  "url": "https://example.com",
  "instruction": "Extract the page title and summary.",
  "schema": {
    "type": "object",
    "properties": {
      "title": {"type": ["string", "null"]},
      "summary": {"type": ["string", "null"]}
    },
    "required": ["title", "summary"],
    "additionalProperties": false
  }
}

MCP browser example

Browser interaction is session based.

First call:

browser_create_session

Save the returned session_id.

Then call:

browser_navigate

with:

{
  "session_id": "<session-id>",
  "url": "https://example.com"
}

Then an action:

browser_action

{
  "session_id": "<session-id>",
  "action": "click",
  "description": "More information"
}

Read page text:

browser_text

{
  "session_id": "<session-id>"
}

Finally:

browser_close_session

{
  "session_id": "<session-id>"
}

Generic MCP client configuration

For MCP clients that accept a Streamable HTTP URL, configure the server URL as:

http://127.0.0.1:8081/mcp

The exact client configuration syntax varies by application. The key point is that the client connects directly to that URL; it does not connect to Firecrawl itself.

7. Choosing REST vs MCP

Use REST when:

• calling from application code

• using deterministic LangGraph nodes

• integrating through FastAPI/httpx/curl

• you want explicit HTTP request/response control

• you want OpenAPI documentation

Use MCP when:

• an LLM or agent should discover tools dynamically

• using an MCP-capable client

• you want the model to choose among search, scrape, extract, and browser tools

• you want one tool server rather than hard-coded HTTP nodes

Both interfaces reach the same underlying services.

8. LangGraph integration

For deterministic LangGraph nodes, point all web-research HTTP calls at:

http://<docker-host>:8080

Do not let individual nodes depend directly on Firecrawl's URL. For example:

search node   -> POST /v1/search
map node      -> POST /v1/map
scrape node   -> POST /v1/scrape
crawl node    -> POST /v1/crawl
extract node  -> POST /v1/extract
interact node -> /v1/interact/...

For an MCP-driven agent, connect the MCP client to:

http://<docker-host>:8081/mcp

This separation lets you replace a backend later without rewriting the graph.

9. Security model

Gateway API key

GATEWAY_API_KEY protects the REST gateway when set.

Generate one with:

openssl rand -hex 32

Do not commit .env.

Network exposure

By default only these ports are published, both on loopback:

127.0.0.1:8080 -> REST gateway
127.0.0.1:8081 -> MCP server

The following remain private to the Compose network:

• Firecrawl API

• Firecrawl Playwright service

• Extract service

• Interact service

• Redis

• RabbitMQ

• PostgreSQL

SSRF protection

The interaction service validates top-level navigation and browser subrequests. Private, loopback, link-local, multicast, and other non-public destinations are rejected unless explicitly permitted by code/policy.

This matters because a browser automation service without SSRF controls can otherwise be used to probe internal services.

Consequential actions

The interaction service applies a conservative text-based policy to clicks that appear consequential. A potentially consequential click returns an error unless the request explicitly sets:

{
  "allow_consequential": true
}

An agent should set this only after the application has obtained explicit user approval for that action.

10. Health and troubleshooting

Gateway

curl -sS http://127.0.0.1:8080/health | jq

If you enabled GATEWAY_API_KEY and your middleware requires it for health in a future version, add the Authorization header.

Verify Ollama from the extract container

docker compose exec extract python - <<'PY'
import urllib.request
print(urllib.request.urlopen('http://host.docker.internal:11434/api/tags', timeout=5).read()[:500])
PY

Verify SearXNG from Firecrawl

docker compose exec firecrawl-api node -e \
  "fetch('http://host.docker.internal:8088/search?q=test&format=json').then(r=>{console.log(r.status);return r.text()}).then(t=>console.log(t.slice(0,500))).catch(e=>{console.error(e);process.exit(1)})"

Search returns no results

Check SearXNG directly:

curl -sS 'http://127.0.0.1:8088/search?q=test&format=json' | jq '.results | length'

Then inspect Firecrawl logs:

docker compose logs --tail=200 firecrawl-api

Confirm:

docker compose exec firecrawl-api env | grep '^SEARXNG_'

Extraction fails

Confirm Ollama sees the model:

curl -sS http://127.0.0.1:11434/api/tags | jq -r '.models[].name'

Then:

docker compose logs --tail=200 extract

Browser fails to start

docker compose logs --tail=200 interact

The interact container uses a Playwright image/runtime and a 1gb shared-memory allocation.

Request correlation

The gateway creates/preserves an x-request-id and forwards it to downstream services. Use that identifier when correlating gateway, extract, and interact logs.

11. Tests

The project includes both unit/regression tests and live end-to-end test scripts. The live tests exercise the actual running REST gateway, Firecrawl, external SearXNG, external Ollama, Playwright interaction service, and MCP server.

Test prerequisites

Start the stack first:

docker compose up -d --build

The live shell tests require curl, python3, and Docker. They automatically read GATEWAY_API_KEY from .env when it is not already exported.

By default the end-to-end tests use:

REST gateway:  http://127.0.0.1:8080
MCP endpoint:  http://127.0.0.1:8081/mcp
Public test URL: https://example.com
Search query:   example domain

Override them when needed:

GATEWAY_URL=http://127.0.0.1:8080 \
MCP_URL=http://127.0.0.1:8081/mcp \
TEST_URL=https://www.iana.org \
SEARCH_QUERY='IANA reserved domains' \
./scripts/test-all.sh

Unit and regression tests

Run the Python tests in the project image:

make test-unit

This checks:

• loopback/private/link-local SSRF blocking

• rejection of non-HTTP URL schemes

• rejection of credentials embedded in URLs

• mixed public/private DNS resolution, which protects against DNS-rebinding-style destinations

• browser host allowlist enforcement

• consequential-action detection

• gateway API-key enforcement

• x-request-id generation and preservation

You can also use the shorter alias:

make test

Live security tests

make test-security

or:

./scripts/test-security.sh

The live security test creates a real isolated Playwright session and verifies that these destinations are rejected through the public REST contract:

127.0.0.1
10.0.0.0/8
192.168.0.0/16
169.254.169.254
file:// URLs
credential-bearing URLs

It also confirms that a bad gateway API key receives HTTP 401 when authentication is enabled, and that unknown browser-session IDs receive 404. It then runs the unit security regression suite inside the gateway image.

Live REST functionality tests

make test-functionality

or:

./scripts/test-functionality.sh

This performs actual end-to-end calls for:

The extraction test supplies content directly instead of scraping it first. This deliberately isolates the Ollama extraction path so a Firecrawl failure cannot hide an Ollama problem.

If you only want local/non-network functionality checks, skip the Firecrawl tests that require outbound Internet and SearXNG:

SKIP_NETWORK_TESTS=1 ./scripts/test-functionality.sh

MCP smoke test

make test-mcp

or:

./scripts/test-mcp.sh

The script executes a FastMCP client from inside the running MCP container. It connects to the real Streamable HTTP endpoint, performs MCP initialization, lists tools, verifies that all expected tools are exposed, and calls the about tool.

Expected tools include:

about
search
map_site
scrape
crawl
crawl_status
extract
browser_create_session
browser_navigate
browser_action
browser_text
browser_screenshot
browser_close_session

Run everything

make test-all

or:

./scripts/test-all.sh

A successful run ends with:

ALL TESTS PASSED

These tests are intentionally kept separate: unit tests are fast and deterministic, while live tests prove that the deployed services and your existing Ollama/SearXNG endpoints actually work together.

12. Production recommendations

Before exposing the stack outside a trusted host/network:

1. Pin Firecrawl and base image versions.

2. Put REST and MCP behind a TLS reverse proxy.

3. Add strong authentication at the reverse proxy/MCP boundary.

4. Keep Redis, RabbitMQ, PostgreSQL, Playwright, Extract, and Firecrawl internal.

5. Add resource limits appropriate to your hardware.

6. Back up persistent Firecrawl state.

7. Add log aggregation and metrics.

8. Define browser-session concurrency based on RAM/CPU capacity.

9. Keep the Interact service's SSRF restrictions enabled.

10. Do not allow an agent to set allow_consequential=true without explicit authorization.

11. Test upgrades in a staging Compose project before changing production.

12. Keep Ollama and SearXNG independently managed so this stack can be upgraded without replacing them.

13. Capability summary

Once running, you have a single self-hosted web-research layer with both REST and MCP access:

                  Web Research Stack
                         |
      +------------------+------------------+
      |                  |                  |
   Discovery          Content          Interaction
      |                  |                  |
 search/map       scrape/crawl/extract     browser
      |                  |                  |
   SearXNG        Firecrawl + Ollama     Playwright

Application code can therefore depend on the Web Research Stack contract rather than directly on Firecrawl, SearXNG, Ollama, or Playwright.