sourcesQL

Web research you can query, inspect and keep working with.

sourcesQL is an open-source research engine and MCP App. A connected assistant turns your request into SQL, searches the web, builds a structured dataset and presents the finished table in chat. You can inspect its sources, correct a field, branch into a follow-up investigation and review a later refresh without losing the original result.

Use it when the answer needs rows: a vendor comparison, a project pipeline, an organization inventory or a collection of claims with supporting sources. The result stays available for further SQL instead of becoming another table trapped in a conversation.

How research works

The assistant plans the search terms. sourcesQL executes the query and keeps the data, evidence and execution receipt.

1. Discover broadly. Use complementary search terms across terminology, source types and angles. Search results are candidate pages, not verified entities.

2. Reduce with SQL. Merge overlapping results, deduplicate sources, filter available fields and join saved datasets before spending on semantic processing.

3. Extract selectively. Apply semantic matching, extraction, scoring or normalization to candidates that still need it, using explicitly configured models.

4. Present the finished table. Keep intermediate work behind the scenes. Open the final selection with readable labels, useful source links and an intentional row order.

5. Continue from saved research. Reopen, query, correct, expand or refresh a result without repeating discovery unnecessarily.

SQL filters run before semantic work where the admitted query plan permits it. A final LIMIT bounds returned rows; it does not make earlier search or model work cheaper.

This query runs against the included fictional fixtures, with no API keys or network calls:

SELECT
  SEM_EXTRACT(content, 'project name explicitly stated') AS project,
  SEM_EXTRACT(
    content,
    'planned capacity in MW, number only',
    type := 'DOUBLE'
  ) AS planned_capacity_mw,
  url AS source_url
FROM WEB_SEARCH('fictional Aurora Birch campuses')
WHERE SEM_MATCH(content, 'describes a specific data-center campus')
ORDER BY project, source_url;

The engine supports WEB_SEARCH, WEB_FETCH, SEM_MATCH, SEM_EXTRACT, SEM_EXTRACT_ALL, SEM_SCORE and SEM_NORM. It accepts an explicit subset of DuckDB-style SELECT and nonrecursive WITH queries. See the SQL reference for signatures, supported staging and refusals. Operator availability also depends on the selected provider profile.

Related MCP server: mcp-research

A workspace for the result

The MCP App adds an interactive table to compatible clients, including ChatGPT. Other MCP clients can use the same tools without rendering the app.

• Saved research: typed columns, stable row identities, retained sources and immutable engine results.

• Corrections: edit a value with a separate revision history; the original engine result remains intact. Follow-up SQL can explicitly use the original or a pinned corrected view.

• Linked investigations: expand a row or field into a child research table, then navigate its tree and return to the parent.

• Refresh review: compare proposed updates and accept or reject them explicitly.

• Portable research: export and restore bundles with their results, corrections and linked history.

The app decorates column labels and source addresses without rewriting data. Missing values stay missing. Search excerpts remain identified as excerpts, and a supporting quote establishes what a source says rather than proving the claim true.

Choose an installation

Every path starts in fixture mode. Live search and model calls require credentials, an enabled trusted profile and explicit execution budgets.

Local Docker

Clone this repository and run from its root with Docker running:

git clone https://github.com/JohnnyFiv3r/sourcesQL.git
cd sourcesQL
docker compose up --build -d sourcesql-app

Open localhost:3117. The initial workspace uses synthetic data. Its named volume preserves results and corrections across restarts.

To connect a local MCP client over stdio, first build the app image:

docker compose build sourcesql-mcp

Then configure the client to run docker compose run --rm -T --no-deps sourcesql-mcp from this checkout. The Docker guide includes a complete MCP configuration, storage details and the CLI container. The default stdio container has no network; live providers require deliberate network and credential configuration.

A local endpoint alone is not reachable by hosted ChatGPT. Use the hosted path below for a remotely accessible, authenticated connection.

Python and CLI

Install uv and Python 3.12–3.14, then run from the repository root:

uv sync --frozen
uv run sourcesql --store local/cli capabilities --json
uv run sourcesql --store local/cli demo --profile fixture --output artifacts/demo
uv run sourcesql --store local/cli query --file examples/01_discovery.sql

Each successful query returns an immutable result_id, typed columns, a bounded row preview and an execution receipt. Read later pages or export the full result:

uv run sourcesql --store local/cli read <result_id> --offset 100 --limit 100
uv run sourcesql --store local/cli export --result <result_id> --output result-bundle.json
uv run sourcesql --store local/restored import result-bundle.json --no-network

For Python integration, import Engine from sourcesql.engine. The examples cover discovery, saved-result refinement, row expansion and snapshot comparison.

To run the app from source, install Node.js 24, keep the root Python environment above, then run:

cd apps/research
npm ci
npm run build
npm start

Open localhost:3117, or use npm run stdio with a local MCP client. See the example client configuration. CLI and app stores are separate by default; use the MCP query/result-reference workflow for app research, or research bundles to move it between stores.

Actual SQL execution requires the supported OS protections: macOS sandbox-exec, or Linux Landlock and seccomp. Missing protection causes a refusal. Use Docker on Windows. See runtime qualification for the tested environments.

Personal cloud: Vercel + MotherDuck

This path uses your own accounts and one owner workspace:

ChatGPT / MCP client
        │
        ▼
Vercel: MCP tools, app UI, authentication
        │
        ▼
MotherDuck: saved research, evidence, corrections, jobs
        │
        └── on-demand Flight: Python research execution

Ordinary reads, tree navigation and corrections use stored records without starting Python. Research runs as a bounded asynchronous job. One job runs at a time per owner namespace; queued jobs resume explicitly. A lost response is recovered using the same request and job IDs.

Setup is manual today:

1. Build and provision a reviewed execution bundle with the MotherDuck setup guide. Submission starts disabled.

2. Create a Vercel project rooted at apps/research, then configure the gateway and encrypted secrets using the Vercel guide.

3. For hosted ChatGPT, configure your OAuth provider and sole owner identity. The documented path uses an adopter-owned Auth0 tenant, a public PKCE client and sourcesql:access. Follow OAuth setup, then connect your stable HTTPS /mcp URL in ChatGPT.

4. Verify fixture execution and saved-result reopening before enabling live providers.

Eligible personal, non-commercial adopters can choose Vercel Hobby; no Pro-only feature is required by this configuration. MotherDuck, search and model consumption are separate. Included allowances may cover light use, but application budgets do not guarantee that a platform account with billing enabled stops before overage.

Configure live search and models

Start with the provider guide and profile template. Keep real configuration and secrets outside version control.

For search-only work, omit model and embedding settings. An Exa API key does not enable semantic operators or authorize spending. Request budgets must fit the trusted profile; source fetching also requires approved destination hosts.

The currently qualified hosted profile, exa-search.v1, uses Exa Auto and retained excerpts. It permits two searches and ten candidate pages per job, with no extra fetches or model calls. It is a small search-only configuration, not broad entity-research capacity. Larger fanout and semantic research need separately configured and qualified execution profiles. See the hosted policy and Exa research notes.

Status and development

The project includes the engine, local Docker app, durable MotherDuck jobs and storage, and hosted ChatGPT OAuth/app rendering. Fixture tests, live provider quality and host integration are different checks; the current status records their scope and remaining gaps. Broad live research quality, hosted semantic models and Claude rendering are not qualified by the small Exa integration.

Run local checks from the repository root:

uv run pytest -q
npm ci --prefix apps/research
npm --prefix apps/research run check

Positive SQL worker tests need permission to launch the OS sandbox. A sandbox refusal is not a passing execution test. See qualification and app integration for the execution, evidence and storage contracts.

Licensed under Apache 2.0. Attribution and source reuse are recorded in NOTICE and provenance.