# Strata (`strata-mcp` & `strata` CLI)

**Multi-layer web search, Hindsight TEMPR ranking, deterministic token-budget packing (`max_tokens`), serverless PDF parsing, and JS SPA extraction for AI Agents.**

[![License: MIT](https://img.shields.io/badge/License-MIT-0b7a6b.svg)](LICENSE)
[![TypeScript](https://img.shields.io/badge/TypeScript-5.7-0f172a)](https://www.typescriptlang.org/)
[![Node](https://img.shields.io/badge/Node-%3E%3D20-15803d)](https://nodejs.org/)
[![MCP](https://img.shields.io/badge/MCP-12_Tools-d94e1f)](https://modelcontextprotocol.io/)

> **Agent Self-Onboarding:** Are you an AI coding agent (Claude Code, OpenCode, Cursor, Codex)? Read [`SKILL.md`](SKILL.md) or [`llms.txt`](llms.txt) to mint a free `2,500`-credit sandbox key and start querying in one command.

![Strata 4-Layer Stratigraphic Architecture](assets/architecture-strata.svg)

---

## Why Strata? (The 3 Structural Advantages Over Tavily, Exa & Firecrawl)

Teams building production AI agents today are forced to stitch together multiple expensive APIs—and pay a heavy **2x credit penalty** every time an agent extracts relevant passages from search results.

Strata consolidates **Search + Passage Highlights + PDF/Markdown Extraction + Site Mapping + Similarity Discovery** into one deterministic engine accessible via **4 channels**:
1. **MCP Server (`stdio` & Remote HTTP):** 12 native tools for Claude Code, Cursor, OpenCode, and Windsurf.
2. **Zero-Token Bash CLI (`npx strata ...`):** Saves ~`3,000` system-prompt tokens per turn compared to loading MCP schemas at rest.
3. **1-Line Drop-In Tavily Replacement (`POST /compat/tavily/search`):** Accepts standard Tavily SDK payloads (`api_key`, `query`, `search_depth: "advanced"`) while charging **1 flat credit instead of 2**.
4. **REST API (`POST /v1/search`, `/v1/extract`, `/v1/map`, `/v1/similar`):** Built for LangChain, Mastra, LlamaIndex, and n8n workflows.

---

## 1. Cost & Credit Efficiency Comparison (Verified October 2026)

Tavily and Firecrawl double your credit usage (`2 credits`) whenever your agent requests search with extracted passages (`search_depth: "advanced"`). Because Strata runs URL-aware **Okapi BM25** and **Hindsight TEMPR** ranking on our own CPU layer, **passage highlights are included for 1 flat credit**.

![Monthly Cost Comparison for 100,000 Agent Searches](assets/cost-comparison.svg)

| Monthly Agent Workload | **Strata** (1x Flat Credit) | **Tavily API** (`$0.005–$0.008/cr`) | **Exa AI** (`$7/1k` + Deep) | **Firecrawl** (2 cr / search) | **Your Savings with Strata** |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Free Tier Allowance** | **`2,500` credits / mo** | `1,000` credits / mo | `$10` credit (~`1,428` searches) | `1,000` credits (`500` searches) | **2.5× more free searches** |
| **25,000 Searches + Highlights** | **$19 / mo** *(Starter Plan)* | **$250 – $400 / mo** *(50k credits)* | **$175 / mo** | **$99 / mo** *(Standard Plan)* | **Save 81% – 95%** |
| **100,000 Searches + Highlights** | **$49 / mo** *(Team Plan)* | **$1,000 – $1,600 / mo** *(200k credits)* | **$700 / mo** | **$399 / mo** *(Growth Plan)* | **Save 88% – 96%** |
| **400,000 Searches + Highlights** | **$149 / mo** *(Scale Plan)* | **$4,000+ / mo** *(800k credits)* | **$2,800 / mo** | **$749 / mo** *(Scale Plan)* | **Save 80% – 96%** |

*Sources verified 2026-10-15 (Grade A primary documentation): [`docs.tavily.com/documentation/api-credits`](https://docs.tavily.com/documentation/api-credits), [`exa.ai/pricing`](https://exa.ai/pricing), [`firecrawl.dev/pricing.md`](https://www.firecrawl.dev/pricing.md).*

---

## 2. Live E2E Latency, Multilingual Precision & Token Efficiency

All benchmarks below are reproducible and versioned in [`benchmarks/search-quality/runs/2026-10-15-phase2-serper-tempr/live-results.json`](benchmarks/search-quality/runs/2026-10-15-phase2-serper-tempr/live-results.json).

![Live E2E Latency and Multilingual Precision@5 Comparison](assets/latency-and-accuracy.svg)

### Head-to-Head Benchmark Ledger

| Task ID | Workload / Query | Tavily MCP | Strata v1.5 (Google SERP + TEMPR) | Key Engineering Takeaway |
| :--- | :--- | :--- | :--- | :--- |
| **`HTTP`** | **Repeat Agent Query (`POST /v1/search`)**<br>Normalized `SHA-256` L1/L2 `SerpCache` | `1,200+ ms` (billed every call) | **`2 ms`** (`cache_hit: true`, `0` upstream calls) | **-99.8% latency** when multiple agents or turns query the same topic. |
| **`s06`** | **Hindsight TEMPR Token Budget (`max_tokens: 600`)**<br>`Python asyncio TaskGroup exception handling` | Unsupported (fixed top-K) | **`1,078 ms`** (`docs.python.org` at **#1**, `342 / 600` tokens packed) | Packs best-first into `max_tokens`, skipping oversized chunks to fit concise ones behind them (**60–80% fewer LLM input tokens**). |
| **`s01`** | **Technical Search EN + 3-Page Highlights**<br>`Python asyncio TaskGroup exception handling documentation` | `1,940 ms` (`5/5`, `docs.python.org` at #4) | **`1,926 ms`** (`5/5`, `anyio` #1, `docs.python.org` **#2** with 2 BM25 chunks) | URL-aware BM25 + official `docs.*` boost keeps canonical documentation above forum noise. |
| **`s02`** | **Academic Paper Search + PDF Highlights**<br>`Reciprocal Rank Fusion Cormack Clarke Buettcher SIGIR 2009` | `1,196 ms` (`5/5`) | **`1,929 ms`** (`5/5`, `research.google` &amp; `dl.acm.org` in top 3) | Also extracts `cormacksigir09-rrf.pdf` (`9,482` chars) in `64 ms` via serverless `unpdf`. |
| **`s03`** | **Multilingual Technical Docs (Italian)**<br>`Google Search Central sitemap robots.txt documentazione italiano` | **`0/5` (`1,709 ms`)**<br>*Returned 5 K-Pop band (TXT) pages* | **`5/5` (`2,340 ms`)**<br>*#1 `developers.google.com/.../robots/intro?hl=it`* | Explicit locale routing (`gl: "it", hl: "it"`) + domain preference beats Tavily `5/5 vs 0/5`. |
| **`s04`** | **Negative Control (Invented Product)**<br>`"Velrion QZ-9274" "official documentation"` | **5 False Positives** (`1,462 ms`, unrelated manuals) | **`0` Results (`777 ms`)**<br>`status: "empty"` | Clean abstention prevents downstream LLM hallucinations on non-existent entities. |
| **`s05`** | **Real-Time Google News (`tbs: qdr:w`)**<br>`Model Context Protocol AI agents` (`category: "news"`) | N/A | **`747 ms`** (`5/5` dated news articles) | Sub-800ms fresh news retrieval via `/news` + `tbs` time windows. |

### How Hindsight TEMPR Saves 60–80% of Your LLM Input Tokens

Adapted from [`vectorize-io/hindsight`](https://github.com/vectorize-io/hindsight), Strata replaces naive character truncation with three deterministic scoring stages:
1. **URL-Aware Okapi BM25 (`k1 = 1.5, b = 0.75`):** Tokenizes title (`2x` weight), URL host/path slugs (`docs`, `python`, `asyncio`), and body text, applying a `1.35x` boost to official documentation hosts (`docs.*`, `developers.*`, `*.readthedocs.io`).
2. **Multiplicative Recency × Proof-Count Boost:**
   $$\text{final\_score} = \text{score} \times \bigl(1 + 0.2(\text{recency} - 0.5)\bigr) \times \bigl(1 + 0.1(\text{proof\_norm} - 0.5)\bigr)$$
   where $\text{proof\_norm} = \text{clamp}(0.5 + \ln(\text{providers.length})/10, 0, 1)$ rewards URLs corroborated by multiple search engines without letting a fresh but irrelevant page leapfrog a high-relevance match.
3. **Token Budget Packing (`max_tokens`):** Packs extracted passages best-first into your exact context window budget (`max_tokens: 600`), skipping an oversized chunk to fit higher-density shorter chunks ranked behind it.

---

## 3. Quickstart (10 Seconds)

### Step 1: Mint a Free Sandbox API Key (`2,500` Credits/Month, No Card)

```bash
curl -s -X POST https://api.strata.sh/v1/keys/free \
  -H "Content-Type: application/json" \
  -d '{"label": "github-quickstart"}'
```

### Step 2: Connect via Your Preferred Channel

#### Channel A — MCP Server (`Claude Code`, `Cursor`, `OpenCode`, `Windsurf`)

Add to `~/.config/opencode/opencode.jsonc` or `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "strata": {
      "command": "npx",
      "args": ["-y", "strata-mcp"],
      "env": {
        "STRATA_API_KEY": "st_free_YOUR_API_KEY",
        "STRATA_API_URL": "https://api.strata.sh"
      }
    }
  }
}
```

#### Channel B — Zero-Token Bash CLI (`0` System-Prompt Overhead)

```bash
export STRATA_API_KEY="st_free_YOUR_API_KEY"
export STRATA_API_URL="https://api.strata.sh"

# 1. Search with Hindsight TEMPR token budget cap
npx strata-mcp search "Python asyncio TaskGroup exception handling" --limit 5 --max-tokens 600

# 2. Extract HTML, /llms.txt, or PDF files
npx strata-mcp extract "https://docs.python.org/3/library/asyncio-task.html"

# 3. Map a documentation site (<sitemapindex> + 4-worker concurrent BFS)
npx strata-mcp map "https://modelcontextprotocol.io" --limit 20

# 4. Find externally similar pages (Entity-Overlap tanh + BM25)
npx strata-mcp similar "https://docs.python.org/3/library/asyncio-task.html" --limit 5

# 5. Calculate monthly workload savings vs Tavily, Exa & Firecrawl
npx strata-mcp cost --searches 25000 --extracts 5000 --research 500
```

#### Channel C — 1-Line Drop-In Replacement for Tavily SDKs (`POST /compat/tavily/search`)

Already using Tavily? Change only the endpoint URL to `https://api.strata.sh/compat/tavily/search` and pay **1 credit instead of 2** on `search_depth: "advanced"`:

```bash
curl -X POST https://api.strata.sh/compat/tavily/search \
  -H "Content-Type: application/json" \
  -d '{
    "api_key": "st_free_YOUR_API_KEY",
    "query": "Google Search Central robots.txt documentazione italiano",
    "search_depth": "advanced",
    "max_results": 5
  }'
```

---

## 4. All 12 MCP Tools & Credit Weights

| MCP Tool | REST Endpoint | Credit Cost | Description |
| :--- | :--- | :--- | :--- |
| `growth_search_cascade` | `POST /v1/search` | **1 credit** | Multi-engine search, RRF fusion, BM25/semantic reranking, Hindsight TEMPR boosts, `1.8s` highlights &amp; `max_tokens` packing. |
| `growth_search_social` | `POST /v1/search` | **1 credit** | Site-scoped search across X (`x.com`), Reddit, GitHub, and Hacker News. |
| `growth_find_similar` | `POST /v1/similar` | **1 credit** | Seed URL profiling + lexical cosine + BM25 + Entity-Overlap `tanh` + `/v1/embeddings`. |
| `growth_map` | `POST /v1/map` | **1 credit** | `/robots.txt` + `/sitemap.xml` (`<sitemapindex>`) + 4-worker BFS + `Crawl4AI` JS SPA fallback. |
| `growth_extract_content` | `POST /v1/extract` | **1 credit** | Single-URL extraction across HTML, `/llms.txt`, `.md`, and `.pdf` (`unpdf`) with 7-day SHA-256 cache. |
| `growth_extract_structured` | Local / MCP | **1 credit** | DOM table &amp; row extraction with deterministic `where` filtering (`price > 100 && name ~ /^pro/i`). |
| `growth_analyze_content_gap` | Local / MCP | **1 credit** | Deterministic SERP domain-diversity score (`0–100`) for SEO/AEO topic analysis. |
| `growth_get_latest` | Local / MCP | **1 credit** | Reads the cached discovery report from disk with zero network calls. |
| `growth_extract_batch` | Local / MCP | **2–3 credits** | Parallel extraction for up to 20 URLs (`2 cr` $\le 10$ URLs, `3 cr` $11\text{–}20$ URLs) with BM25 passage selection. |
| `growth_research` | Local / MCP | **3 cr (`mini`) / 5 cr (`pro`)** | 2-hop adaptive research: parallel Hop-1 sub-queries, emergent term extraction for Hop 2, RRF fusion &amp; cited report. |
| `growth_run_discovery` | Local / MCP | **3 credits** | Queries 10 authoritative AI/engineering sources and saves `latest-report.json`. |
| `growth_crawl` | Local / MCP | **5 credits** | Combines `growth_map` discovery with 3-worker parallel content extraction and highlight selection. |

---

## 5. Self-Hosted Local Mode (Optional Docker Stack)

If you prefer to run Strata 100% locally with `deedy5/ddgs` (Rust `primp` TLS impersonation) and `Crawl4AI`:

```bash
git clone https://github.com/alexgenovese/strata.git
cd strata
npm install && npm run build
docker compose up -d
STRATA_MODE=local npm test
```

## License

MIT © [Alex Genovese](https://github.com/alexgenovese)