Evident

An open-source, agent-agnostic extraction and fetch layer.

Turn any URL — or any question about the web — into verified, typed data with a transparent confidence score, callable by any AI agent through MCP, REST, or a native SDK.

Evident is not another scraper. Best-in-class open-source scraping/rendering engines already exist (Crawl4AI, Playwright). Evident orchestrates them behind a resilience ladder, scores every result's trustworthiness, and lets you extract structured data from any site — not just ones someone hand-wrote a parser for — via a versioned, community-contributable recipe system.

Full vision, architecture, and roadmap: docs/VISION.md.

Why

Most extraction tools give you clean text and let you figure out whether to trust it. Evident's whole design centers on one missing piece: every result carries a confidence score and a method explaining how it was produced, so an autonomous agent — not a human — can decide whether to act on it.

What confidence actually measures today: which code path produced a result (recipe match vs. LLM extraction vs. raw fetch, official API vs. reverse-engineered, fully-rendered vs. partial content) — a provenance/method signal, not a correctness signal. It is deterministic and internally consistent (a recipe match always outscores raw fetch, for example), but it is not yet calibrated against any ground truth — nothing in the pipeline compares an extracted value to what's actually correct. Treat a 0.9 as "produced by a method that's usually reliable," not as "90% likely to be factually right." Calibration against a held-out benchmark is tracked as future work in docs/VISION.md §9.

Related MCP server: Haunt API

Quickstart

git clone https://github.com/Kaushalendra-Marcus/evident
cd evident
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"

# Run the MCP server (stdio) — works with Claude Desktop, Claude Code, Cursor,
# or any other MCP-compatible client
evident-mcp

Add to your MCP client config (example for Claude Desktop):

{
  "mcpServers": {
    "evident": {
      "command": "/absolute/path/to/.venv/bin/evident-mcp"
    }
  }
}

Not using an MCP client? Same engine, plain Python:

import asyncio
from evident.core import ladder

async def main():
    result = await ladder.run("https://example.com")
    record = ladder.to_record(result)
    print(record.confidence, record.method)
    print(record.data.get("markdown", "")[:500])

asyncio.run(main())

Tools (MCP) / functions (SDK)

Optional dependencies

Evident's core (Tier 1 static fetch) has minimal dependencies on purpose. Heavier capabilities are opt-in:

pip install "evident[render]"   # Tier 2: JS-rendered pages via Crawl4AI/Playwright
pip install "evident[llm]"      # extract(): LLM-based schema extraction (bring your own ANTHROPIC_API_KEY)
pip install "evident[api]"      # REST API interface
pip install "evident[all]"      # everything, plus dev/test tooling

If render isn't installed and Tier 1 fails, fetch() reports failure_reason: dependency_missing instead of crashing — Tier-1-only installs stay fully usable for the large share of the web that's server-rendered.

Contributing a recipe

The easy on-ramp for contributors is a recipe, not the core engine. A recipe is one YAML metadata file plus one small async fetcher function — see docs/RECIPE_GUIDE.md.

Testing

pip install -e ".[dev]"
pytest

Tests use recorded/mocked HTTP responses (respx) so they run deterministically without live network access — this was the single biggest gap in earlier hand-rolled scraping projects this one grew out of, and it's non-negotiable here.

Status

Early / pre-1.0. Tier 1 (static fetch) and the recipe registry (Greenhouse, Lever, Ashby) are implemented and unit-tested against mocked fixtures. Tier 2 (rendered fetch via Crawl4AI) is implemented and has been smoke-tested against a live page. LLM-based extract() is implemented but requires your own ANTHROPIC_API_KEY and hasn't been live-tested end-to-end yet — see docs/VISION.md roadmap for what's next.

Security

Evident's whole job is server-side fetching of caller-supplied URLs — treat it accordingly. As shipped, there is no authentication, no SSRF protection, and no rate limiting on the REST API or the Docker image (docker-compose.yml publishes port 8000 directly). This is a deliberate scope cut for this stage, not an oversight, but it means:

• Do not expose the REST API directly to the public internet. Run it behind your own reverse proxy with auth (or keep it on localhost/a private network) if you use evident.api.rest or the Docker image.

• Do not point fetch()/extract() at untrusted, caller-controlled URLs in a multi-tenant deployment without adding your own SSRF guard (deny private/link-local IP ranges and cloud metadata endpoints like 169.254.169.254) in front of it — nothing in the current codebase does this for you.

• The MCP server (stdio, single local user) doesn't have this exposure in the same way and is the lowest-risk way to run Evident today.

See the DD report / roadmap for planned hardening. Found a security issue? Please open a GitHub issue rather than a public PR with exploit details.

License

Apache-2.0 — see LICENSE. Deliberately not AGPL, to stay commercial-use-friendly.