encore
README.md
# encore
**Personalized music search and recommendation, measured.** Seven recommenders and a
learning-to-rank search stack, run on real Last.fm listening data. The evaluation is
leak-free and reports confidence intervals and significance tests. The winning stack is
served over HTTP and as MCP tools that an LLM agent can call.
[](https://github.com/Brennan-Richards/encore/actions/workflows/ci.yml)
## Headline results
| Question | Answer (held-out test data, 3 random splits) |
|---|---|
| Does personalizing search help? | **NDCG@10 0.524 vs 0.376** for the same LambdaMART ranker without personal features: **+40.7%** [95% CI +38.0%, +43.4%], p < 1e-260, 5,493 queries |
| vs. a typical "lexical + popularity" search baseline? | **+110%** NDCG@10 (0.524 vs 0.248) |
| Where does it help most? | Broad queries such as "rock" or "electronic": **0.211 vs 0.039** NDCG@10 (5.4×) |
| Best recommender? | LightGCN with the friend graph, NDCG@10 **0.279 ± 0.005**. EASE reaches 98% of that and trains about 100× faster (0.6 s vs 63 s) |
| Do friends help? | In recommendation, slightly: +1.7% [+0.4%, +3.1%], p = 0.03. In search, no: −0.1%, p = 0.94 |
| Is it fast enough to serve? | Full search stack (retrieve 200 → 12 features → LambdaMART): **p50 0.79 ms, p99 1.44 ms** per query on a laptop CPU |
## What's here
```mermaid
flowchart LR
D[Last.fm HetRec 2011<br/>plays · tags · friends] --> S[per-user 70/10/20 split<br/>×3 seeds]
S --> R[7 recommenders<br/>MostPop · ItemKNN · EASE · iALS<br/>BPR-MF · LightGCN · LightGCN+Social]
S --> T[tag index + BM25<br/>held-out tag events removed]
R -->|train-only CF scores| F[12 ranking features]
T --> F
F --> L[LambdaMART<br/>LightGBM lambdarank]
L --> E[Engine]
E --> A[FastAPI]
E --> M[MCP server<br/>5 agent tools]
R & L --> V[metrics · bootstrap CIs<br/>paired Wilcoxon · slices · SHAP]
```
- **`src/encore/models.py`**: seven recommenders behind one `fit` / `score` interface, written
from the papers in NumPy and PyTorch. Covers neighbourhood, closed-form linear,
confidence-weighted MF, pairwise MF, and graph convolution with an optional social graph.
- **`src/encore/search.py`**: discovery search. Word-level BM25 retrieval over artist tag
documents, 12 features (query match, popularity, three CF scores, user taste vs. artist
tags, user interest in the query, friends' listening), and LambdaMART re-ranking.
- **`src/encore/metrics.py`**: NDCG / Recall / MRR, coverage, popularity bias, bootstrap CIs,
and paired significance tests.
- **`src/encore/engine.py`, `api.py`, `mcp_server.py`**: the benchmarked model served as is.
Every search result can be explained with per-feature TreeSHAP contributions.
- **`tests/`**: 20 offline tests on a synthetic catalog, including the leakage guards. CI also
smoke-runs both benchmarks on the real dataset.
- **[`docs/METHODOLOGY.md`](docs/METHODOLOGY.md)** and **[`docs/REFERENCES.md`](docs/REFERENCES.md)**: the task definitions, leakage controls, credits,
and limitations, written out in full.
## Results
All numbers are produced by the scripts and stored in [`results/`](results/). The
full tables, including every paired test and the chosen hyperparameters, are in
[`results/RESULTS.md`](results/RESULTS.md).
### 1. Personalized discovery search
A listener types a genre or mood and wants artists **new to them**. Queries are top tags of
each user's held-out artists. Every ranker sees the same BM25 top-200 candidates, which
contain 81% of the relevant artists.
| Ranker | Personal signal | NDCG@10 (mean ± std, 3 splits) | Recall@10 | MRR@10 |
|---|---|---|---|---|
| BM25 | — | 0.138 ± 0.003 | 0.264 | 0.103 |
| BM25 + popularity (tuned) | — | 0.248 ± 0.002 | 0.399 | 0.214 |
| CF only (EASE score) | ✓ | 0.296 ± 0.004 | 0.463 | 0.272 |
| BM25 + popularity + EASE (tuned linear blend) | ✓ | 0.352 ± 0.003 | 0.519 | 0.327 |
| LambdaMART, non-personalized features only | — | 0.376 ± 0.006 | 0.550 | 0.340 |
| LambdaMART, no CF features | ✓ | 0.486 ± 0.004 | 0.651 | 0.459 |
| LambdaMART, no social features | ✓ | 0.523 ± 0.007 | 0.678 | 0.502 |
| **LambdaMART, all features** | ✓ | **0.524 ± 0.007** | **0.680** | **0.502** |
Comparing full LambdaMART with non-personalized LambdaMART separates the value of
personalization from the value of switching to a learned ranker. Both use the same model
class and training data; only the personal features differ.
<p float="left">
<img src="figures/search_by_breadth.png" width="49%" />
<img src="figures/search_feature_importance.png" width="49%" />
</p>
Personalization matters most where the query alone says least. For **broad** tags shared by
more than 123 artists, non-personalized ranking is close to guessing (NDCG@10 0.04–0.09) and
the personal ranker reaches 0.21. The SHAP breakdown shows the same picture. After the
query-tag match, the ranker relies on ItemKNN similarity to the user's history, on how well
the user's taste matches the artist's tags, and on iALS. The friend-graph features carry
almost no weight, and dropping them changes nothing measurable.
**The same query from two listeners**, served by the trained model (`Engine.search`):
| `"post-rock"` | Top 5 |
|---|---|
| Anonymous (popularity-aware fallback) | Isis · Mogwai · 65daysofstatic · God Is an Astronaut · Interpol |
| User 2 (Depeche Mode, Daft Punk, Goldfrapp, Gorillaz…) | Sigur Rós · M83 · Broken Social Scene · The American Dollar · The Notwist |
The personalized list moves toward the electronic and dream-pop side of post-rock, which
fits that listener's history.
### 2. Recommendation
| Model | NDCG@10 | Recall@20 | Coverage@10 | Fit time |
|---|---|---|---|---|
| **LightGCN + friend graph** | **0.279 ± 0.005** | **0.358** | 0.50 | 63 s |
| LightGCN | 0.276 ± 0.003 | 0.351 | 0.37 | 51 s |
| EASE | 0.273 ± 0.001 | 0.347 | 0.29 | 0.6 s |
| iALS | 0.265 ± 0.004 | 0.346 | 0.33 | 2.7 s |
| ItemKNN | 0.264 ± 0.001 | 0.336 | 0.43 | 0.1 s |
| BPR-MF | 0.224 ± 0.006 | 0.296 | 0.46 | 1.7 s |
| MostPop | 0.093 ± 0.003 | 0.123 | 0.01 | — |
<img src="figures/recsys_cost_quality.png" width="70%" />
The ranking is a cost/quality decision, not only a leaderboard. LightGCN+Social wins on
accuracy and covers 50% of the catalog in top-10 lists, against 29% for EASE. EASE gets 98%
of the accuracy in under a second with no tuning beyond one regularizer. That is why the
served search stack uses EASE as one of its CF features, alongside iALS and ItemKNN.
## What's established vs. what's new here
Every model, metric and protocol in this repo comes from published work, credited in
[`docs/REFERENCES.md`](docs/REFERENCES.md). No new algorithm is claimed. What this repo
adds is a set of controlled measurements on one public dataset:
**New measurements.** We did not find these reported elsewhere for HetRec Last.fm.
1. **The size of the personalization effect, isolated from the ranker.** On identical
candidates, with the same learner and training queries, adding personal features lifts
discovery-search NDCG@10 by **+40.7%**. The gain over the classic BM25 + popularity
baseline is +110%, or 0.276 NDCG. The controlled comparison splits that gain: about half
(0.148) comes from the personal features, and the rest from learning to rank at all.
2. **Collaborative filtering still adds value inside search once taste profiles are
present.** Removing the three CF features (keeping tag-based taste) costs 7.3%
(p ≈ 1e-42). Tag-based user profiles alone, the classic folksonomy approach, leave
measurable accuracy on the table.
3. **The friend graph helps recommendation slightly but adds nothing to search.**
+1.7% for LightGCN with vs. without friend edges (p = 0.03 on split 0; positive on all
three splits, but close to zero on one). Exactly 0 in search (−0.1%, p = 0.94) once CF features are present. In this
data, friends' signal appears to be already captured by co-listening.
**Results that confirm prior findings** (credited, not claimed as discoveries):
- Personalization pays off most on broad queries (5.4× on broad tags vs. 1.7× on narrow
ones). This matches Teevan, Dumais & Liebling (SIGIR 2008): personalization helps most
where intent varies across users.
- A well-tuned linear model (EASE) gets within 2% of the best graph neural model at about 1%
of the training cost. This matches Ferrari Dacrema et al. (RecSys 2019) and Steck (2019).
## How the evaluation avoids fooling itself
1. Artist tag profiles drop every tag event on a held-out (user, artist) pair, so a user's
own tags cannot leak into queries or features.
2. All search features come from models fit on the training split only.
3. Artists the user already plays are never candidates, so a model cannot win by
re-surfacing them.
4. Hyperparameter grids, linear-blend weights, LambdaMART early stopping and neural
early stopping all use validation data. Test is scored once per split.
5. Results are reported over three random splits, with 95% bootstrap CIs and paired
Wilcoxon tests for every claim above.
Limitations are spelled out in [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md#known-limitations).
The main ones: the dataset is small (1.8K users, 2.8K artists), queries are tags rather than
logged search strings, and nothing here was tested online.
## Reproduce
```bash
git clone https://github.com/Brennan-Richards/encore && cd encore
python -m venv .venv && source .venv/bin/activate
make install # pip install -e ".[serve,dev]"
make test # 20 offline tests, ~5 s
make all # download (2.5 MB, SHA-256 pinned) → recsys (~17 min) → search (~4 min) → report
```
On an Apple M3 laptop CPU, the full recommendation benchmark (tuning plus three splits)
takes about 17 minutes and the search benchmark about 4. `make smoke` runs both with tiny
grids in about a minute.
## Serve it
```bash
make serve # http://localhost:8000/docs
curl "localhost:8000/search?q=post-rock&user_id=2&k=5"
curl "localhost:8000/explain?q=post-rock&user_id=2&artist=Sigur%20R%C3%B3s"
```
To give Claude (or any MCP client) the tools `search_music`, `recommend_artists`,
`similar_artists`, `user_profile` and `explain_ranking`:
```json
{
"mcpServers": {
"encore": { "command": "/path/to/encore/.venv/bin/python", "args": ["-m", "encore.mcp_server"] }
}
}
```
The served engine is the split-0 model exactly as benchmarked. User profiles therefore show
only training-split artists. Anonymous queries use the tuned BM25 + popularity ranker,
because the personal ranker was never trained on users without history.
## Next steps
- Natural-language queries: an LLM rewrites "something moody for a rainy drive" into
catalog tags, evaluated with an LLM-as-judge against held-out listening.
- Two-tower candidate generation with approximate nearest-neighbour search in place of
BM25-only retrieval, and recall@K measured at each stage.
- Navigational queries (artist names) and a re-finding setting, to complement discovery.
- A larger, timestamped dataset (e.g. LFM-1b-style session logs) for a temporal split.
## Credits
- **Data:** [Last.fm](http://www.last.fm). The HetRec 2011 Last.fm-2k release was built by
Ignacio Fernández-Tobías, Iván Cantador and Alejandro Bellogín (UAM) and is distributed by
[GroupLens](https://grouplens.org/datasets/hetrec-2011/) for non-commercial use. It is
cited as Cantador, Brusilovsky & Kuflik, *HetRec 2011*, RecSys 2011. The data is downloaded
at run time and not redistributed here.
- **Methods:** every model, metric and protocol is cited, with the file that implements it,
in [`docs/REFERENCES.md`](docs/REFERENCES.md).
- **Development:** built with [Claude Code](https://claude.com/claude-code) as an AI
pair-programmer. Commits carry a `Co-Authored-By` trailer.
## License
MIT for the code. The dataset keeps its original terms.