<div align="center">

<img src="public/meta-image.png" alt="Evident — turn messy information into trusted evidence" width="840" />

# Evident

<p><strong>An open-source, agent-agnostic extraction and fetch layer that returns typed data with a transparent <em>confidence score</em> on every result.</strong></p>

<p>Turn any URL — or any question about the web — into verified, typed data, callable by any AI agent through <strong>MCP</strong>, <strong>REST</strong>, or a native <strong>SDK</strong>.</p>

<p>
  <a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License: Apache 2.0" /></a>
  <img src="https://img.shields.io/badge/Python-3.11%20%7C%203.12-blue.svg" alt="Python 3.11 | 3.12" />
  <a href="https://github.com/Kaushalendra-Marcus/evident/actions/workflows/ci.yml"><img src="https://github.com/Kaushalendra-Marcus/evident/actions/workflows/ci.yml/badge.svg" alt="CI status" /></a>
  <img src="https://img.shields.io/badge/status-pre--1.0-orange.svg" alt="Status: pre-1.0" />
</p>

<p>
  <a href="#quickstart"><strong>Quickstart</strong></a> ·
  <a href="#tools-mcp--functions-sdk"><strong>Tools</strong></a> ·
  <a href="docs/RECIPE_GUIDE.md"><strong>Write a recipe</strong></a> ·
  <a href="docs/VISION.md"><strong>Vision</strong></a> ·
  <a href="SECURITY.md"><strong>Security</strong></a>
</p>

</div>

Evident is not another scraper. Best-in-class open-source scraping/rendering engines already exist ([Crawl4AI](https://github.com/unclecode/crawl4ai), Playwright). Evident orchestrates them behind a resilience ladder, scores every result's trustworthiness, and lets you extract structured data from *any* site — not just ones someone hand-wrote a parser for — via a versioned, community-contributable recipe system.

> Full vision, architecture, and roadmap: [`docs/VISION.md`](docs/VISION.md).

## Why

Most extraction tools give you clean text and let you figure out whether to trust it. Evident's whole design centers on one missing piece: **every result carries a `confidence` score and a `method` explaining how it was produced**, so an autonomous agent — not a human — can decide whether to act on it.

> **What `confidence` actually measures today:** which code path produced a result (recipe match vs. LLM extraction vs. raw fetch, official API vs. reverse-engineered, fully-rendered vs. partial content) — a provenance/method signal, not a correctness signal. It is deterministic and internally consistent (a recipe match always outscores raw fetch, for example), but it is **not yet calibrated against any ground truth** — nothing in the pipeline compares an extracted value to what's actually correct. Treat a 0.9 as "produced by a method that's usually reliable," not as "90% likely to be factually right." Calibration against a held-out benchmark is tracked as future work in `docs/VISION.md` §9.

## Quickstart

```bash
git clone https://github.com/Kaushalendra-Marcus/evident
cd evident
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"

# Run the MCP server (stdio) — works with Claude Desktop, Claude Code, Cursor,
# or any other MCP-compatible client
evident-mcp
```

Add to your MCP client config (example for Claude Desktop):

```json
{
  "mcpServers": {
    "evident": {
      "command": "/absolute/path/to/.venv/bin/evident-mcp"
    }
  }
}
```

Not using an MCP client? Same engine, plain Python:

```python
import asyncio
from evident.core import ladder

async def main():
    result = await ladder.run("https://example.com")
    record = ladder.to_record(result)
    print(record.confidence, record.method)
    print(record.data.get("markdown", "")[:500])

asyncio.run(main())
```

## Tools (MCP) / functions (SDK)

| Tool | What it does |
|---|---|
| `fetch(url, mode)` | Universal fetch, escalates the resilience ladder automatically |
| `extract(url, json_schema)` | Structured extraction against **any** caller-supplied schema — works on any site |
| `list_recipes()` | Discover built-in, verified extraction recipes |
| `use_recipe(recipe_id, slug, entity_name)` | Invoke a deterministic, high-confidence recipe (e.g. `ats_greenhouse`) |
| `health_check(target)` | Proactively check whether a recipe or URL is still working |

## Optional dependencies

Evident's core (Tier 1 static fetch) has minimal dependencies on purpose. Heavier capabilities are opt-in:

```bash
pip install "evident[render]"   # Tier 2: JS-rendered pages via Crawl4AI/Playwright
pip install "evident[llm]"      # extract(): LLM-based schema extraction (bring your own ANTHROPIC_API_KEY)
pip install "evident[api]"      # REST API interface
pip install "evident[all]"      # everything, plus dev/test tooling
```

If `render` isn't installed and Tier 1 fails, `fetch()` reports `failure_reason: dependency_missing` instead of crashing — Tier-1-only installs stay fully usable for the large share of the web that's server-rendered.

## Contributing

Contributions are welcome. The single most valuable — and lowest-friction — contribution is a **recipe**: one YAML metadata file plus one small async fetcher function, no need to understand the resilience ladder or confidence engine. See [`docs/RECIPE_GUIDE.md`](docs/RECIPE_GUIDE.md) for the recipe walkthrough, and [`CONTRIBUTING.md`](CONTRIBUTING.md) for the development setup, the two project rules every change must satisfy (a test that would have caught the bug; no unverified "it works" claims), and how to run the checks CI runs.

## Testing

```bash
pip install -e ".[dev]"
pytest
```

Tests use recorded/mocked HTTP responses (`respx`) so they run deterministically without live network access — this was the single biggest gap in earlier hand-rolled scraping projects this one grew out of, and it's non-negotiable here.

## Status

Early / pre-1.0. Tier 1 (static fetch) and the recipe registry (Greenhouse, Lever, Ashby) are implemented and unit-tested against mocked fixtures. Tier 2 (rendered fetch via Crawl4AI) is implemented and has been smoke-tested against a live page. LLM-based `extract()` is implemented but requires your own `ANTHROPIC_API_KEY` and hasn't been live-tested end-to-end yet — see `docs/VISION.md` roadmap for what's next.

## Security

Evident's whole job is server-side fetching of caller-supplied URLs — treat it accordingly. The shipped code protects part of that surface and deliberately leaves the rest to your own deployment.

**What the code protects.** Every fetch path — `fetch()`, `extract()`, `health_check()`, the resilience ladder (Tier 1 and Tier 2), and the recipe fetchers — enforces a built-in SSRF guard. A URL whose host resolves to a non-public address (loopback, RFC1918 private ranges, link-local including the cloud metadata endpoint `169.254.169.254`, and multicast/reserved/unspecified ranges) is refused before any connection is opened, as is any non-`http(s)` scheme. Hostnames are resolved and the resulting IPs are checked — not string-matched — so a DNS-rebinding name that points at an internal address is still blocked. A refused URL returns a diagnosable `ssrf_blocked` result rather than failing silently. See [`src/evident/core/ssrf.py`](src/evident/core/ssrf.py).

**What the code does not protect (by design, for now).** The REST API and Docker image ship with **no authentication and no rate limiting** (`docker-compose.yml` publishes port 8000 directly). This is a deliberate scope decision for a self-hosted, single-operator tool, not an oversight — so:

- **Do not expose the REST API directly to the public internet.** Run it behind your own reverse proxy with auth, or keep it on localhost / a private network, if you use `evident.api.rest` or the Docker image.
- The MCP server (stdio, single local user) has no such network exposure and is the lowest-risk way to run Evident today.

Full security posture and how to report a vulnerability: [`SECURITY.md`](SECURITY.md). **Please report security issues through GitHub's private vulnerability reporting — not a public issue or PR with exploit details.**

## License

Apache-2.0 — see [`LICENSE`](LICENSE). Deliberately not AGPL, to stay commercial-use-friendly.

<div align="center">
  <br />
  <img src="public/logo.png" alt="Evident mascot" width="90" />
  <br />
  <sub><strong>Evident</strong> — how sure should an agent be that a result is correct?</sub>
</div>