# Selvage

[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Node](https://img.shields.io/badge/node-%3E%3D20-brightgreen.svg)](package.json)

**The self-finished edge that keeps your scattered knowledge from unraveling.**

A selvage is the woven edge of fabric that stops it from fraying. Selvage the
project is the same idea for your team's knowledge: a thin, hybrid-searchable
layer over Slack, wikis, code, and databases, queryable by humans, MCP clients,
and agents. Self-hosted, local-first, MIT-licensed.

## Why

Every growing team reinvents the same broken thing: a half-searchable wiki, a
Slack history nobody can find, docs that quietly go stale. The usual fix, "put
everything in one place," never survives contact with how people actually work.
Data keeps living where it's produced.

Selvage takes the other path: a common interface over data *wherever it already
lives*. Connectors normalize each source into one shared shape and land it in a
single Postgres table. Retrieval knows nothing about where a record came from,
so exact error codes and paraphrased questions both work, and the same query
serves a person at a CLI, an agent over MCP, or the web UI.

- **Pluggable connectors.** A small contract, not a fixed integration list.
  Ships with filesystem/git-repo, Slack, and generic Postgres-table connectors.
  Writing your own is about 50 lines ([guide](docs/connector-authoring.md)).
- **One unified store.** Postgres + pgvector. Every source lands in the same
  `documents` table; nothing connector-specific leaks into retrieval.
- **Hybrid retrieval.** Full-text, vector, and rare-term lists, fused with
  Reciprocal Rank Fusion, recency-decayed, then cross-encoder reranked.
- **Local by default.** Embeddings and reranking run on your CPU via ONNX. The
  quickstart needs no API key, no cloud account, no tokens.
- **MCP-native.** Narrow, LLM-free tools any agent can compose, plus a minimal
  web UI that adds a plan → fan-out → synthesis loop for humans.

## Quickstart

Five minutes, needs **Node 20+** and **Docker**.

```bash
git clone https://github.com/naidx0/selvage.git && cd selvage
npm install
npm run setup          # starts Postgres+pgvector in Docker (port 5433), migrates schema
npm run sync           # indexes examples/docs via the filesystem connector
npx tsx cli.ts search "ERR_GRIPPER_TIMEOUT_0x2F"        # exact-token query
npx tsx cli.ts search "how do new engineers get set up" # semantic query
npx tsx cli.ts who-knows "gripper stalls"               # who has written about a topic
```

The first query in a process loads the local models (a few seconds, up to ~20s
cold on a busy machine); the first run ever downloads ~40MB of ONNX models from
the Hugging Face hub and caches them for later. No account required.

## Point it at your own data

The quickstart indexes the sample corpus in `examples/docs`. To index your own,
edit `selvage.config.json` and change the filesystem source's `path` to any
folder of markdown, code, or text, then sync again.

Before (ships in the repo):

```json
{
  "sources": [
    { "id": "example-docs", "connector": "filesystem", "config": { "path": "./examples/docs" } }
  ],
  "projects": {
    "handbook": { "description": "Example project scoping only the sample docs", "sources": ["example-docs"] }
  }
}
```

After (your handbook plus a code repo, bundled into one project):

```json
{
  "sources": [
    { "id": "handbook", "connector": "filesystem", "config": { "path": "/Users/you/notes" } },
    { "id": "app-code", "connector": "filesystem", "config": { "path": "/Users/you/src/app" } }
  ],
  "projects": {
    "engineering": { "description": "Docs + code for the app team", "sources": ["handbook", "app-code"] }
  }
}
```

```bash
npm run sync                              # re-index everything
npx tsx cli.ts sources                    # list sources + document counts
npx tsx cli.ts search "your query" --project engineering
```

A **project** is a named bundle of sources that scopes a query so you don't get
cross-domain noise. A source can live in many projects without being duplicated.
`SELVAGE_CONFIG` points at a different config file if you don't want it in the
repo root.

## Connectors

A connector is one self-contained folder, `connectors/<name>/index.ts`, that
implements a four-field contract: `{ name, syncMode, validateConfig, sync }`.
Core handles embedding, batching, upserts, change detection, pruning, and
project scoping. Three ship in the box.

### filesystem

Points at a directory and indexes every known-text file, chunked at
language-aware boundaries (markdown at headings, code at declaration
boundaries, prose at paragraphs). `full` sync, so deleted and renamed files
drop out of the index on the next run. For markdown it also extracts an author
(YAML frontmatter `author:` or a leading `**Author:**` line) so `who-knows`
works on plain files.

```json
{
  "id": "handbook",
  "connector": "filesystem",
  "config": { "path": "./docs", "exclude": [".git", "node_modules"], "maxFileSizeKb": 512 }
}
```

No secrets. `exclude` and `maxFileSizeKb` are optional (defaults shown).

### slack

Polls `conversations.history` per channel and resumes from a durable cursor,
so it runs fine from a cron job or a one-shot `sync` with just a bot token. It
also walks thread replies. The bot must be a member of each channel.

```json
{
  "id": "eng-slack",
  "connector": "slack",
  "config": { "channels": ["C0123ABC", "C0456DEF"], "lookbackDays": 180 }
}
```

Set the token in the environment (never in the config file):

```bash
export SLACK_BOT_TOKEN=xoxb-...
```

Required bot scopes: `channels:history`, `channels:read`, `users:read`,
`team:read`. Override the env var name with `tokenEnv` if you prefer.

### postgres

The "bring your own database" reference. Point it at a table (or arbitrary
`SELECT`) and map columns onto the shared schema. Keyset pagination, `full`
sync. `examples/postgres-seed.sql` is a ready-made sample table to try it.

```json
{
  "id": "tickets",
  "connector": "postgres",
  "config": {
    "table": "support_tickets",
    "columns": { "id": "id", "content": "body", "title": "subject", "author": "agent", "timestamp": "updated_at" }
  }
}
```

Set the source database URL in the environment:

```bash
export SELVAGE_PG_SOURCE_URL=postgres://user:pass@host:5432/dbname
```

Override the env var name with `connectionStringEnv`. Provide exactly one of
`table` or `query`; the `id` column must be orderable.

### Write your own

The contract is the whole surface, and core never needs touching.
[docs/connector-authoring.md](docs/connector-authoring.md) walks through a
complete JSON-lines connector in about 50 lines, with the rules that keep a
connector honest (secrets from env, fail loudly, pure exported normalization,
log to stderr) and a pre-ship checklist.

## Retrieval architecture

```mermaid
flowchart LR
  subgraph sources [Your data, where it already lives]
    FS[Folder / git repo]
    SL[Slack]
    PG[(Any Postgres table)]
    X[Your connector...]
  end

  subgraph selvage [Selvage]
    C1[Connectors: normalize]
    DB[(Postgres + pgvector<br/>unified documents table)]
    subgraph retrieval [Hybrid retrieval, LLM-free]
      FTS[full-text GIN]
      VEC[vector HNSW]
      RARE[rare-term IDF]
      RRF[RRF fusion k=60]
      DECAY[recency decay]
      RERANK[cross-encoder rerank]
      FTS --> RRF
      VEC --> RRF
      RARE --> RRF
      RRF --> DECAY --> RERANK
    end
    C1 -->|shared schema + embeddings| DB
    DB --> FTS & VEC & RARE
  end

  subgraph clients [Clients own orchestration]
    MCP[MCP server]
    WEB[Web UI]
    CLI[CLI]
  end

  FS & SL & PG & X --> C1
  RERANK --> MCP & WEB & CLI
```

Pure vector search loses on exact tokens like error codes and flag names, and
short-but-rare messages sink under generic filler on cosine similarity. So
Selvage runs three ranked lists per query, each over the resolved project or
source scope:

1. **Full-text** over a stored `tsvector` with a GIN index, ranked by
   `ts_rank_cd`. Wins on exact tokens.
2. **Vector**, pgvector HNSW cosine, the query embedded with the BGE
   instruction prefix. Wins on paraphrase.
3. **Rare-term**, an IDF-style guard: query tokens that are rare in the corpus
   get their own list so a single exact match can't be drowned out.

The three lists fuse with **Reciprocal Rank Fusion**: `score(d) = Σ 1/(60 +
rank)`. RRF needs only ranks, which is exactly right when `ts_rank_cd` and
cosine similarity live on incomparable scales. k=60 is the standard default
from Cormack, Clarke & Büttcher, *"Reciprocal Rank Fusion Outperforms Condorcet
and Individual Rank Learning Methods"* (SIGIR 2009).

A **recency decay** factor in `[0.5, 1]` (half-life 90 days) discounts stale
answers without erasing them, so an exact-token hit on an old postmortem still
surfaces. Finally the fused top-12 goes through a local **cross-encoder**
(`Xenova/ms-marco-MiniLM-L-6-v2`) that reads query and document together. Every
result carries a `signals` object with the per-list ranks, RRF score, decay
factor, and raw rerank logit, so any ranking is debuggable.

Embeddings are local ONNX (`Xenova/bge-small-en-v1.5`, 384-dim, q8) via
`@huggingface/transformers`, on CPU, no API key. Full detail and the decision
log: [docs/architecture.md](docs/architecture.md).

## Demo

Real CLI output against the sample corpus (12 chunks), trimmed to the top hits.

Exact-token query lands on the error-registry chunk, with the matching
postmortem right behind it:

```
$ npx tsx cli.ts search "ERR_GRIPPER_TIMEOUT_0x2F"

1. [0.997] example-docs · Fleet API reference › Error registry (excerpt)
   ## Error registry (excerpt) | Code | Meaning | | `ERR_GRIPPER_TIMEOUT_0x2F` |
   Servo watchdog fired during gripper actuation. | ...
   fts#2  vec#1  rare#2  rrf=0.0487  recency=1.00  rerank=8.11

2. [0.989] example-docs · Postmortem: fleet-wide gripper stalls, 2026-03-12
   # Postmortem: fleet-wide gripper stalls, 2026-03-12 **Severity:** SEV-1 ...
   fts#1  vec#2  rare#1  rrf=0.0489  recency=1.00  rerank=7.95
```

`who-knows` aggregates authored matches across sources:

```
$ npx tsx cli.ts who-knows "gripper stalls"

1. Priya Raman — 2 matches (sources: example-docs, latest 2026-07-17)
```

Add `--json` to any query for machine-readable output, `--project <name>` or
`--sources <a,b>` to scope it, and `--no-rerank` to skip the cross-encoder.

## MCP

Expose retrieval to Claude Code or any MCP client. The tools are deliberately
narrow and LLM-free, so an agent can fan out over them cheaply and owns its own
orchestration.

```bash
# Run from the repo root; the path must be absolute.
claude mcp add selvage -- npx tsx "$(pwd)/mcp-server/index.ts"
```

Tools exposed:

| Tool | Does |
|---|---|
| `selvage_search` | Hybrid search, returns chunks with their citation uri |
| `selvage_who_knows` | Ranks people by how much they authored on a topic |
| `selvage_list_sources` | Lists connected sources with document counts |
| `selvage_list_projects` | Lists projects (search scopes) and their sources |

`selvage_search` and `selvage_who_knows` take an optional `project` or `sources`
scope. Smoke-test the server end-to-end anytime:

```bash
npx tsx scripts/mcp-smoke.mts
```

## Web UI

```bash
npm run web            # http://localhost:3000 (override with SELVAGE_WEB_PORT)
```

**Search mode** is fully local, the same hybrid pipeline as the CLI.
**Ask mode** adds a reference agent loop (plan → parallel tool fan-out →
synthesized, cited answer) over the same tools; it needs `ANTHROPIC_API_KEY`
set. Without the key, Search still works and Ask reports itself as disabled
rather than failing. Model defaults to `claude-sonnet-5`
(`SELVAGE_ANTHROPIC_MODEL`).

The web UI is single-node and has no built-in auth. Run it on localhost or
behind your own proxy.

## Configuration

Everything has a working default except the secrets for optional connectors and
Ask mode. Copy `.env.example` to `.env` or export in your shell.

| Variable | Default | Purpose |
|---|---|---|
| `DATABASE_URL` | `postgres://selvage:selvage@localhost:5433/selvage` | Selvage's own Postgres (matches `docker-compose.yml`) |
| `SELVAGE_CONFIG` | `selvage.config.json` | Path to the source/project config |
| `SELVAGE_EMBEDDINGS` | `local` | `local` (ONNX, no account) or `openai` (any OpenAI-compatible `/v1/embeddings`) |
| `SELVAGE_EMBED_MODEL` | `Xenova/bge-small-en-v1.5` | Local embedding model |
| `SELVAGE_EMBED_DIM` | `384` | Embedding dimension |
| `SELVAGE_EMBED_DTYPE` | `q8` | `q8` (fast) or `fp32`; part of the index identity |
| `SELVAGE_HALF_LIFE_DAYS` | `90` | Recency-decay half-life |
| `SELVAGE_RERANK` | `true` | Cross-encoder rerank of the fused top-N |
| `SELVAGE_RERANK_MODEL` | `Xenova/ms-marco-MiniLM-L-6-v2` | Cross-encoder model |
| `SLACK_BOT_TOKEN` | — | Bot token for a Slack source |
| `SELVAGE_PG_SOURCE_URL` | — | Source DB URL for a Postgres source |
| `ANTHROPIC_API_KEY` | — | Enables Ask mode in the web UI |
| `SELVAGE_ANTHROPIC_MODEL` | `claude-sonnet-5` | Model for Ask mode |
| `SELVAGE_WEB_PORT` | `3000` | Web UI port |

`OPENAI_API_KEY` / `OPENAI_BASE_URL` apply only when `SELVAGE_EMBEDDINGS=openai`.
Full annotated list in [.env.example](.env.example).

## Repo structure

```
core/          shared schema, embeddings, chunking, retrieval (RRF, rerank, recency)
connectors/    one self-contained folder per connector (filesystem, slack, postgres)
mcp-server/    MCP tool exposure (stdio)
web/           minimal reference UI + agent loop
docs/          architecture + connector-authoring guide
examples/      sample corpus + postgres seed to try everything without real data
cli.ts         migrate / sync / search / who-knows / sources / projects
```

## Limitations

Honest about what's proven and what isn't.

- **Scale is unproven at the high end.** Verified on a small corpus. CPU
  embeddings are fine for thousands of documents; hundreds of thousands is
  untested.
- **English-only full-text.** The Postgres FTS config is `'english'`. Other
  languages still get vector and rare-term matching, but not the full-text list.
- **Slack is unit-tested, not live-tested.** The normalization and polling logic
  have tests, but the connector has not run against a real workspace. Polling
  also misses edits and deletions of already-indexed messages (fetches are
  "newer than cursor" only).
- **`access_scope` is stored, not enforced.** It's surfaced on every result but
  never used as a filter. Selvage is a retrieval layer, not an authorization
  system; callers that need enforcement read the label and drop what a user may
  not see.
- **Single-node, no web UI auth.** Run it on localhost or behind your own proxy.
- **No migration versioning yet.** The schema evolves by a fresh `migrate`.

## Roadmap

- **More connectors.** Notion, GitHub, and an Obsidian-vault connector are the
  next targets. Want a different source? Open a
  [connector request](.github/ISSUE_TEMPLATE/connector_request.md).
- **Incremental-sync reconciliation** so incremental connectors (Slack) can pick
  up edits and deletes, not just new records.
- **Non-English full-text configs**, selectable per source.

## How this was built

Selvage was built and finished end-to-end with orchestrated AI agents: parallel
exploration sweeps, phase-gated build waves with hard file-ownership boundaries
so no two agents touched the same file, and three rounds of adversarial external
review that converged to zero findings. A 37-test gate (9 of them DB-backed
integration) stands over every change, and every claim in this README was
executed live before publication. The full receipt, including the plan and what
actually shipped, is in [docs/finish-plan.md](docs/finish-plan.md).

## Contributing

Bug reports, connector requests, and PRs welcome. Start with
[CONTRIBUTING.md](CONTRIBUTING.md); adding a connector is the headline
contribution path.

## License

MIT. Fork it, extend it, ship it. See [LICENSE](LICENSE).