# Market Researcher

[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/)
[![MCP](https://img.shields.io/badge/interface-MCP-5b5bd6)](https://modelcontextprotocol.io/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

Market Researcher is a local research engine for AI coding agents. It collects public discussions, keeps the evidence behind each finding, and gives the host agent a structured way to test claims before turning them into product ideas.

The useful part is the audit trail. A finding can be traced from the final claim back to the exact quote, document, source URL, offsets, and stored content hash that produced it.

```text
research question
    ↓
query expansion
    ↓
source collection
    ↓
relevance + signal extraction
    ↓
exact evidence spans
    ↓
needs and observations
    ↓
claim verification + counterevidence
    ↓
product hypotheses
    ↓
evidence pack
```

## Why I built it

Most AI market-research workflows are good at producing polished summaries. They are much worse at showing why a conclusion should be trusted.

A few recurring problems pushed this project in a different direction:

- search results were being treated as evidence without preserving the original text;
- several quotes from one person could look like several independent confirmations;
- product ideas were easy to confuse with verified market problems;
- research often stopped after an arbitrary number of searches;
- the final answer was hard to audit after the chat ended.

Market Researcher stores the research state locally and keeps those layers separate.

## What it does

The engine can:

- search supported public sources through source-specific connectors;
- normalize and deduplicate documents inside each research run;
- combine lexical and vector retrieval;
- extract typed signals such as problems, workarounds, switching intent, spend signals, objections, and desired outcomes;
- expand later searches from terminology and problems discovered during the run;
- store exact evidence spans with offsets and hashes;
- distinguish direct evidence, derived observations, claims, counterevidence, and product hypotheses;
- verify claims using independent authors and platforms;
- search for evidence that weakens a claim instead of collecting only support;
- track source coverage, marginal yield, and stopping reasons;
- export deterministic evidence packs for completed runs;
- expose the same engine through a CLI and an MCP server.

The host model handles semantic judgment. The local engine handles persistence, provenance, retrieval, accounting, run isolation, and export integrity.

## Architecture

```mermaid
flowchart LR
    A[Host agent<br/>Codex / Claude] --> B[Agent Skill]
    B --> C[MCP server]
    C --> D[Research engine]

    D --> E[Source connectors]
    D --> F[Query lattice]
    D --> G[Signal extraction]
    D --> H[Claim verification]
    D --> I[Counterevidence]

    E --> J[Normalized documents]
    J --> K[SQLite]
    J --> L[Local vector index]

    G --> M[Evidence spans]
    H --> N[Claim dispositions]
    M --> O[Evidence pack]
    N --> O
```

## Evidence model

The project does not use one opaque "opportunity score." Different kinds of evidence stay separate.

| Layer | Meaning |
|---|---|
| Direct evidence | Exact text stored from a source document |
| Derived observation | An interpretation supported by evidence |
| Claim | An observation submitted to verification |
| Counterevidence | Evidence that contradicts or narrows a claim |
| Product hypothesis | A forward-looking idea that still needs market validation |

A claim can have several supporting quotes and still remain unresolved when they come from too few independent authors or platforms.

See [docs/METHODOLOGY.md](docs/METHODOLOGY.md) and [docs/EVIDENCE_MODEL.md](docs/EVIDENCE_MODEL.md).

## Installation

Requirements:

- Python 3.12+
- `uv`
- Git

```bash
git clone https://github.com/autkucakan/market-researcher.git
cd market-researcher

uv sync --all-extras --dev
cp .env.example .env

uv run market-research doctor
```

`doctor` checks local storage, the embedding/index setup, source configuration, and MCP startup.

## Use it from the CLI

```bash
uv run market-research research \
  "Find recurring operational problems that small software teams repeatedly work around."
```

Useful inspection commands include:

```bash
uv run market-research runs
uv run market-research inspect-run <run_id>
uv run market-research coverage <run_id>
uv run market-research search "agent memory"
uv run market-research evidence <span_id>
```

## Use it from Codex

Register the local MCP server:

```bash
codex mcp add market-research -- \
  "$(command -v uv)" run \
  --directory "$(pwd)" \
  market-research mcp
```

Then call the Agent Skill from Codex:

```text
$market-research

Find startup ideas based on recurring problems people complain about online.
Focus on problems a small technical team could realistically solve and where
people already show behavioral demand through spending, switching, or maintained
workarounds.

Do not use paid sources.
```

The Skill drives the research workflow. You do not need to micromanage internal MCP lifecycle calls in the prompt.

## Evidence packs

A completed run is exported under the configured data directory:

```text
runs/<run_id>/
├── report.md
├── evidence.jsonl
├── claims.json
├── sources.csv
└── manifest.json
```

The newest successfully completed run is available through:

```text
runs/latest/
```

The export keeps the relationship between claims and the underlying evidence. Depending on the source, records include:

- evidence and document IDs;
- exact quotes;
- signal types;
- canonical URLs;
- author identifiers;
- publication and collection times;
- exact offsets;
- stored document hashes;
- verification dispositions;
- supporting or contradictory relationships.

Re-exporting unchanged persisted state is deterministic. `manifest.json` records hashes for the exported files.

## Zero-cost research

`MARKET_RESEARCH_DEFAULT_MAX_BUDGET_USD=0.00` means paid operations are forbidden. It does not mean the research run should stop immediately.

Free connectors can continue until another stopping condition is reached. A positive monetary budget stops only when recorded chargeable usage reaches that budget.

Source availability still depends on credentials and upstream API rules.

## Source families

The connector layer supports research across sources such as:

- GitHub
- Hacker News
- Stack Exchange
- Discourse
- YouTube
- RSS / Atom
- Bluesky
- Mastodon
- Reddit
- X
- generic web and forums

Your environment may expose only a subset. `market-research doctor` and source-status tools report what is configured, unavailable, unattempted, attempted with zero results, or blocked by the current budget.

## Reproducibility

The system keeps enough state to review a completed run at three levels.

**Run level:** brief, queries, source accounting, iteration yield, stopping reason.

**Claim level:** disposition, supporting evidence, counterevidence, verification notes.

**Evidence level:** exact quote, offsets, document hash, source, URL, timestamps.

A new live run is not expected to reproduce the same corpus byte for byte because external sources change. The persisted evidence pack for an unchanged completed run is designed to be reproducible.

See [docs/REPRODUCIBILITY.md](docs/REPRODUCIBILITY.md).

## Testing

Run the current suite locally:

```bash
uv run pytest -q
uv run python -m build
```

The project tests behavior that matters to the research record, including run isolation, evidence-span integrity, claim state transitions, budget semantics, counterevidence handling, and evidence export.

The five stateful MCP regression tests cover research creation and state retrieval, source-status reporting, exact-quote extraction and evidence retrieval, relevance persistence, and counterevidence handling. Run them directly with:

```bash
uv run pytest -q \
  tests/mcp/test_mcp_tools.py::test_mcp_create_research_and_state \
  tests/mcp/test_mcp_tools.py::test_mcp_source_status \
  tests/mcp/test_mcp_agent_workflow.py
```

These MCP tests automatically use a temporary SQLite database, an in-memory Qdrant index, and a temporary export directory. They do not read or modify the configured local research store, and they can run while a local MCP server owns the persistent Qdrant lock.

## Limits

This is an opportunity-discovery tool. It does not prove product-market fit.

A run can still be biased by source availability, platform demographics, search coverage, historical posts, inaccessible private communities, host-model classification errors, or a document limit reached before saturation.

The report should say why research stopped and which claims remain unresolved. Product ideas remain hypotheses until they are tested with prospective users.

## Security

Keep connector credentials in environment variables. Do not commit `.env`, local databases, vector stores, or evidence packs that contain material you do not intend to publish.

See [SECURITY.md](SECURITY.md).

## Contributing

Changes are welcome when they preserve the research record and make the system easier to audit.

See [CONTRIBUTING.md](CONTRIBUTING.md).

## Citation

Citation metadata is available in [`CITATION.cff`](CITATION.cff).

## License

MIT. See [LICENSE](LICENSE).