Evident

Evident is not another scraper. Best-in-class open-source scraping/rendering engines already exist (Crawl4AI, Playwright). Evident orchestrates them behind a resilience ladder, scores every result's trustworthiness, and lets you extract structured data from any site — not just ones someone hand-wrote a parser for — via a versioned, community-contributable recipe system.

Full vision, architecture, and roadmap: docs/VISION.md.

Why

Most extraction tools give you clean text and let you figure out whether to trust it. Evident's whole design centers on one missing piece: every result carries a confidence score and a method explaining how it was produced, so an autonomous agent — not a human — can decide whether to act on it.

What confidence actually measures today: which code path produced a result (recipe match vs. LLM extraction vs. raw fetch, official API vs. reverse-engineered, fully-rendered vs. partial content) — a provenance/method signal, not a correctness signal. It is deterministic and internally consistent (a recipe match always outscores raw fetch, for example), but it is not yet calibrated against any ground truth — nothing in the pipeline compares an extracted value to what's actually correct. Treat a 0.9 as "produced by a method that's usually reliable," not as "90% likely to be factually right." Calibration against a held-out benchmark is tracked as future work in docs/VISION.md §9.

Related MCP server: Haunt API

Quickstart

git clone https://github.com/Kaushalendra-Marcus/evident
cd evident
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"

# Run the MCP server (stdio) — works with Claude Desktop, Claude Code, Cursor,
# or any other MCP-compatible client
evident-mcp

Add to your MCP client config (example for Claude Desktop):

{
  "mcpServers": {
    "evident": {
      "command": "/absolute/path/to/.venv/bin/evident-mcp"
    }
  }
}

Not using an MCP client? Same engine, plain Python:

import asyncio
from evident.core import ladder

async def main():
    result = await ladder.run("https://example.com")
    record = ladder.to_record(result)
    print(record.confidence, record.method)
    print(record.data.get("markdown", "")[:500])

asyncio.run(main())

Tools (MCP) / functions (SDK)

Optional dependencies

Evident's core (Tier 1 static fetch) has minimal dependencies on purpose. Heavier capabilities are opt-in:

pip install "evident[render]"   # Tier 2: JS-rendered pages via Crawl4AI/Playwright
pip install "evident[llm]"      # extract(): LLM-based schema extraction (bring your own ANTHROPIC_API_KEY)
pip install "evident[api]"      # REST API interface
pip install "evident[all]"      # everything, plus dev/test tooling

If render isn't installed and Tier 1 fails, fetch() reports failure_reason: dependency_missing instead of crashing — Tier-1-only installs stay fully usable for the large share of the web that's server-rendered.

Contributing

Contributions are welcome. The single most valuable — and lowest-friction — contribution is a recipe: one YAML metadata file plus one small async fetcher function, no need to understand the resilience ladder or confidence engine. See docs/RECIPE_GUIDE.md for the recipe walkthrough, and CONTRIBUTING.md for the development setup, the two project rules every change must satisfy (a test that would have caught the bug; no unverified "it works" claims), and how to run the checks CI runs.

Testing

pip install -e ".[dev]"
pytest

Tests use recorded/mocked HTTP responses (respx) so they run deterministically without live network access — this was the single biggest gap in earlier hand-rolled scraping projects this one grew out of, and it's non-negotiable here.

Status

Early / pre-1.0. Tier 1 (static fetch) and the recipe registry (Greenhouse, Lever, Ashby) are implemented and unit-tested against mocked fixtures. Tier 2 (rendered fetch via Crawl4AI) is implemented and has been smoke-tested against a live page. LLM-based extract() is implemented but requires your own ANTHROPIC_API_KEY and hasn't been live-tested end-to-end yet — see docs/VISION.md roadmap for what's next.

Security

Evident's whole job is server-side fetching of caller-supplied URLs — treat it accordingly. The shipped code protects part of that surface and deliberately leaves the rest to your own deployment.

What the code protects. Every fetch path — fetch(), extract(), health_check(), the resilience ladder (Tier 1 and Tier 2), and the recipe fetchers — enforces a built-in SSRF guard. A URL whose host resolves to a non-public address (loopback, RFC1918 private ranges, link-local including the cloud metadata endpoint 169.254.169.254, and multicast/reserved/unspecified ranges) is refused before any connection is opened, as is any non-http(s) scheme. Hostnames are resolved and the resulting IPs are checked — not string-matched — so a DNS-rebinding name that points at an internal address is still blocked. A refused URL returns a diagnosable ssrf_blocked result rather than failing silently. See src/evident/core/ssrf.py.

What the code does not protect (by design, for now). The REST API and Docker image ship with no authentication and no rate limiting (docker-compose.yml publishes port 8000 directly). This is a deliberate scope decision for a self-hosted, single-operator tool, not an oversight — so:

• Do not expose the REST API directly to the public internet. Run it behind your own reverse proxy with auth, or keep it on localhost / a private network, if you use evident.api.rest or the Docker image.

• The MCP server (stdio, single local user) has no such network exposure and is the lowest-risk way to run Evident today.

Full security posture and how to report a vulnerability: SECURITY.md. Please report security issues through GitHub's private vulnerability reporting — not a public issue or PR with exploit details.

License

Apache-2.0 — see LICENSE. Deliberately not AGPL, to stay commercial-use-friendly.