Intently

Describe it badly. Find it anyway.

Intently is an intent-first search engine for libraries of tools: AI models, apps, code libraries, MCP servers, or anything else people look for by what it does rather than what it is called. People type vague, misspelled, rambling requests; AI agents describe the problem they are stuck on. Intently works out what they mean, searches six different ways at once, has an LLM check the shortlist, and answers with the tools that fit, why they fit, and where to get them.

It ships as a small Python package with a web UI, a REST API, a Model Context Protocol (MCP) server for agents (stdio and Streamable HTTP), and a one-command AWS Lambda deployment. It runs on a laptop CPU: no GPU, no vector database, no search cluster.

Live demo: ALFRED's AI TOOL BELT, a library of about 4,900 AI tools, models and research projects built from 418 videos of the AI Search YouTube channel (2023 to 2026). Agents can use it directly: claude mcp add --transport http alfreds-tool-belt https://alfreds.org/api/toolbelt/mcp

Contents

• What it can do

• Quick start

• Use it from AI agents

• How it works

• Why this architecture

• Results

• Benefits

• Drawbacks and limits

• Build your own library

• Configuration

• Deployment

• Evaluation

• Project layout

Related MCP server: Hermes Atlas MCP Server

What it can do

Every result carries evidence: a deep link to the exact video moment where the tool is shown, or the project page for catalog entries, plus official links (GitHub, Hugging Face, project sites) matched automatically.

Without any LLM keys it still works in offline mode (hybrid retrieval only), just with less understanding of vague requests.

This is what an agent receives from find_tools on the live demo (markdown format, trimmed):

# Tools for: Create a 3D model of a product using a few phone photos

## 1. 3D reconstruction from photos
- **AnyRecon** (tool app) | fit: exact | Open-source, local, free; handles multiple unaligned smartphone photos
  AnyRecon is a lightweight 3D reconstruction tool that takes a series of unaligned photos taken at random angles ...
  open source: yes · runs locally: yes · pricing: free · availability: released
  Links: https://yutian10.github.io/AnyRecon/
  Seen in: Robot girlfriends, recursive AI agents, full AI research, Happy Horse: AI NEWS (2026-05-03) https://www.youtube.com/watch?v=7r_WJ9xpne0&t=1022s
- **ShapeR** (model · Meta) | fit: exact | Open-source, runs locally, free; reconstructs from photo sequences
  ...

Quick start

Requires Python 3.10+.

git clone https://github.com/Alphaxalchemy/intently
cd intently
pip install -e .

Build the example library (29 well-known open-source AI tools; no LLM or API key needed):

intently ingest catalog examples/catalog
intently build

The first build downloads the embedding model (about 90 MB). Then:

intently serve                    # web UI, API and MCP at http://127.0.0.1:8423
intently search "pull the vocals out of a song"
intently find-tools "transcribe a two person podcast with speaker labels, locally"

For smart mode (query understanding and LLM reranking), copy .env.example to .env and add at least one key. Free tiers work: Groq or Gemini. Anthropic, OpenAI (or any OpenAI-compatible endpoint), Kimi and DeepSeek are also supported.

Use it from AI agents

Intently is designed to be called by an agent in the middle of a task: "before I build this, is there a tool for it?"

MCP, remote (Streamable HTTP):

claude mcp add --transport http intently http://127.0.0.1:8423/mcp

MCP, local (stdio):

claude mcp add intently -- intently mcp

Any MCP client works the same way: point it at /mcp over HTTP, or run intently mcp as a stdio server.

REST:

curl "http://127.0.0.1:8423/api/find-tools?task=remove+background+noise+from+podcast+audio&format=markdown"
curl "http://127.0.0.1:8423/api/search?q=wisper+but+faster"
curl "http://127.0.0.1:8423/api/entity/fasterwhisper"

Every instance also serves /llms.txt (a plain-text guide for agents) and /docs (OpenAPI). AGENTS.md is the full guide for agents, both for using Intently and for working on this code base, and skills/tool-belt/SKILL.md is a ready-made agent skill that tells an agent when and how to ask the live Tool Belt for tools.

How it works

flowchart LR
  subgraph Index["Index time (once per new content)"]
    S1[Video transcripts] --> N[Normalize: documents, sections, passages]
    S2[Catalog entries YAML/JSON] --> N
    N --> E[LLM card per section: names and aliases, summary, capabilities, attributes, 6 search phrasings, relations]
    E --> R[Entity registry: aliases, versions, relations, official links]
    E --> B[BM25: cards and raw passages]
    E --> D[Dense vectors: cards, synthetic questions, raw passages]
  end
  subgraph Query["Query time"]
    Q[Request] --> A[1. Understand: intent, rewrites, hypothetical answer, names, constraints]
    A --> C[2. Retrieve: 6 channels x every query variant]
    C --> F[3. Fuse: weighted reciprocal rank fusion + constraint boosts]
    F --> K[4. Rerank: LLM grades top candidates 0-3]
    K --> G[5. Group by tool, attach links and evidence]
  end
  R -.-> C
  B -.-> C
  D -.-> C

Index time. Each source is normalized into documents (a video, a catalog entry), sections (a chapter, an entry) and passages (about 220 words). For video transcripts, chapter start times are recovered from the description timestamps, and caption fragments that spill across chapter boundaries are moved back. An LLM then writes one structured card per section: canonical names plus aliases (including speech-recognition misspellings such as "RD" for "Ardy"), a plain summary, capabilities in the words users search with, attributes (open source, runs locally, pricing, license, minimum VRAM, availability), six realistic search queries and relations to other tools. Catalog entries already contain most of this, so they need no LLM. From the cards Intently builds:

• an entity registry, a deliberately light graph: aliases merged, version families grouped ("Wan 2.1, 2.2, 2.5"), relations counted, official links matched from video descriptions;

• BM25 indexes over card fields and over raw passages, for exact names and version numbers;

• dense vector indexes over cards, over every synthetic question, and over raw passages, computed locally with MongoDB/mdbr-leaf-ir (23M parameters) through ONNX Runtime.

Query time.

1. Understand (one fast LLM call, about 1 s). The request becomes an intent sentence, three rewrites in the vocabulary a product page or transcript would use, a hypothetical answer (HyDE), corrected product names and constraints (open source, local, free, VRAM, approximate dates). Without an LLM, a heuristic repairs typos against the corpus vocabulary and extracts constraints with patterns.

2. Retrieve (about 100 ms, all local). Six channels run for every query variant: dense cards, dense synthetic questions, dense raw passages, BM25 cards, BM25 passages, and entity lookup (fuzzy alias match, then every version of the named thing).

3. Fuse. Weighted reciprocal rank fusion (k = 20, weights grid-tuned), then soft boosts and penalties for the stated constraints and for intro, outro and sponsor segments.

4. Rerank (one or two LLM calls, 1 to 2 s). A listwise call grades the top 24 candidates 0 to 3 against the intent, seeing each candidate's summary and its best-matching raw excerpt. A second pool is graded only if no exact match was found. Recall safety: anything the reranker did not confirm still follows as "unverified", so a lazy or wrong reranker cannot hide the right answer.

5. Group sections by their primary tool and attach deep links, official links and evidence.

find_tools adds a planning step: one LLM call splits the agent's task into one to four capabilities and writes search variants for each. Each capability then goes through retrieval and reranking in parallel, on different rate-limit buckets, under a latency deadline. A capability whose rerank is late keeps its fused ranking, marked unverified, instead of stalling the agent.

Every stage degrades gracefully, and every response says which stages ran and which model did the work.

Why this architecture

The problem has a specific shape. Queries are short or rambling descriptions of one item. The words in the query often share nothing with the words in the source ("movement animation" vs "text-to-motion"). Names are misspelled twice over, by users and by speech recognition. And the library is full of near-duplicates (versions, forks, rivals). Each popular approach solves part of this:

Intently combines the pieces that address each failure, following published evidence (details and sources in docs/research):

• Write the index in the users' language at index time. LLM-written cards and synthetic questions close the vocabulary gap from the document side (the idea behind Anthropic's Contextual Retrieval, doc2query and QuOTE). On our hardest queries, dense search over cards alone found the right tool in the top 3 about 50% more often than dense search over raw text (0.32 vs 0.21).

• Rewrite the query too. Intent sentences, rewrites and a hypothetical answer (HyDE) close the gap from the query side; this lifted hybrid retrieval from 0.27 to 0.49 hit@3 on the hard set.

• Keep several signals and fuse them. Dense, lexical and entity channels fail on different queries. Reciprocal rank fusion is robust and needs no score calibration; its weights were tuned on a dev split and accepted only because they also improved the held-out split.

• Let an LLM make the final call, but never let it delete. Listwise LLM reranking is where vague intent and constraints are actually resolved (0.49 to 0.73 hit@3 on the hard set). Recall safety keeps its mistakes visible.

• Use a graph only where a graph helps. The entity registry handles aliases, versions, relations and per-tool pages without GraphRAG's indexing cost or its blending of similar tools.

• Stay small. A 23M-parameter embedder on CPU, numpy arrays instead of a vector database, and at most a few thousand tokens of LLM calls per query: cheap enough for free API tiers and a serverless function.

Results

Measured on the live demo's library (4,973 chapters from 418 YouTube AI-news videos), on a held-out test split. Two query sets: easy (101 lazy, typo-ridden, rambling or half-remembered queries written from raw chapter text by a different model family than the one that wrote the cards, plus real user queries) and hard (71 queries machine-checked to share zero content words with the target chapter). "Hit@k" counts the right tool anywhere in the top k, in any video.

An LLM judge that saw only raw captions (never the cards) graded the top 3 of each system 0 to 3:

Latency: about 2 to 3 s per smart search or find_tools call with Groq's free tier, about 0.1 s in offline mode. The engine loads this index in about 2 s and uses about 400 MB of RAM.

Benefits

• Finds things that keyword and plain vector search cannot, especially when the request shares no words with the answer (0.73 vs 0.01 to 0.27 hit@3 above).

• Cheap to run. CPU only. The whole index is flat files (about 180 MB for 5,000 sections) loaded into memory. Query cost is a few thousand LLM tokens, which free tiers cover for personal and small-team use.

• Degrades instead of failing. No LLM keys, a rate-limited vendor or a lazy reranker all produce weaker but still useful results, and the response says which stages ran.

• Explains itself. Each result carries a fit grade, a reason, the understood intent and constraints, and evidence links.

• Agent-native. MCP over stdio and stateless HTTP (works behind proxies and on serverless), compact markdown output, and a tool (find_tools) shaped around how agents actually ask.

• Easy to grow. Add a YAML entry or a transcript, run ingest, enrich and build; enrichment and embeddings are cached, so only new content costs anything.

Drawbacks and limits

• Index-time LLM cost. About one call per six sections (the demo library took about 1,000 calls). Changing the card format means re-enriching.

• Best quality needs an LLM at query time. Offline mode is noticeably weaker on vague requests (0.27 vs 0.73 hit@3 on the hard set). Free tiers cap daily volume, and beyond the cap a public instance falls back to offline ranking.

• Latency. 2 to 3 seconds in smart mode, versus about 100 ms offline.

• Exact in-memory search. Right for thousands to roughly a hundred thousand sections; much larger libraries need an approximate nearest-neighbor index and more memory.

• Tuned for tools. The default profile (categories, VRAM and license attributes, prompts) targets AI and software tools. Other domains need a new profile, and possibly different card fields and re-tuned fusion weights.

• LLM-written metadata can be wrong. Cards are instructed to say "unknown" rather than guess, raw passages remain a separate signal, and official links are matched heuristically. Check before relying on an attribute.

• Evaluated on one library. The numbers above come from one corpus and synthetic plus real queries; expect different numbers elsewhere. The evaluation harness is included so you can measure your own.

• English-centric heuristics. Typo repair and stemming assume English; the LLM stages handle other languages.

Build your own library

Two source types ship with Intently. Both can live in the same library.

Catalog entries (YAML or JSON, one file or a folder). Only name and summary are required:

- name: Basic Pitch
  developer: Spotify
  type: model
  url: https://github.com/spotify/basic-pitch
  summary: Lightweight neural network that transcribes audio of singing or instruments into MIDI notes.
  capabilities: [audio to MIDI, polyphonic note transcription]
  categories: [music_audio]
  open_source: yes
  runs_locally: yes
  pricing: free
  license: Apache-2.0
  install: pip install basic-pitch
  usage: basic-pitch <output-dir> <audio-file>
  alternatives: [Melodyne]

See intently/sources/catalog.py for every field and examples/catalog/ai-tools.yaml for 29 examples.

Video transcripts (Markdown, one *_Transcript.md per video, with a description block and ### chapter headings), the format written by common YouTube transcript downloaders. See intently/sources/transcripts.py.

intently ingest catalog path/to/catalog        # repeatable; --name keeps several catalogs apart
intently ingest transcripts path/to/transcripts
intently enrich                                # LLM cards for new sections (cached, needs a key)
intently build                                 # indexes; only new text is embedded

Profiles hold everything corpus-specific: prompt wording, the category list and labels, example queries, UI text and MCP instructions. Copy intently/profiles/ai-tools.json, edit it, and point INTENTLY_PROFILE at it before building. The index keeps a copy of the profile it was built with.

Configuration

Everything is an environment variable (a .env file in the working directory is read automatically).

Deployment

• Any machine: intently serve --host 0.0.0.0 --port 8423 (FastAPI + uvicorn). Set the rate limits above for a public instance.

• AWS Lambda: deploy/aws-lambda builds a Linux package without Docker and deploys a CloudFormation stack: a Function URL, the index in S3 (copied to /tmp on cold start, about 2.5 s), API keys in SSM Parameter Store, and a warm-up schedule. At low traffic the cost is close to zero. This is how the live demo runs, behind a Next.js route on alfreds.org that forwards /api/toolbelt/* with a shared secret.

• Inside an existing website: deploy/static_page.py renders the web UI as one static HTML file that calls the API under any path prefix, so a site can host the page and proxy the API (how alfreds.org/toolbelt is served).

Evaluation

Put query files in <data dir>/eval/queries_<set>.json:

[{"query": "that thing that turns a song into notes", "answer_entity": "Basic Pitch", "target": "tool-basic-pitch#0", "style": "vague_capability"}]

intently eval                                       # ablation ladder, hit@k and MRR per system
python -m intently.eval.judge --systems=hybrid,full # LLM-graded nDCG (use a different model family than the reranker)
python -m intently.eval.tune --llm                  # grid-tune fusion weights on dev, accept only if test improves too

Files named queries_hard*.json are reported separately. Re-run the evaluation after any change to retrieval.

Project layout

License

MIT, see LICENSE. Created by Alfred Kolakkal.