encore

Personalized music search and recommendation, measured. Seven recommenders and a learning-to-rank search stack, run on real Last.fm listening data. The evaluation is leak-free and reports confidence intervals and significance tests. The winning stack is served over HTTP and as MCP tools that an LLM agent can call.

Headline results

Related MCP server: mcp-spotify

What's here

flowchart LR
    D[Last.fm HetRec 2011<br/>plays · tags · friends] --> S[per-user 70/10/20 split<br/>×3 seeds]
    S --> R[7 recommenders<br/>MostPop · ItemKNN · EASE · iALS<br/>BPR-MF · LightGCN · LightGCN+Social]
    S --> T[tag index + BM25<br/>held-out tag events removed]
    R -->|train-only CF scores| F[12 ranking features]
    T --> F
    F --> L[LambdaMART<br/>LightGBM lambdarank]
    L --> E[Engine]
    E --> A[FastAPI]
    E --> M[MCP server<br/>5 agent tools]
    R & L --> V[metrics · bootstrap CIs<br/>paired Wilcoxon · slices · SHAP]

• src/encore/models.py: seven recommenders behind one fit / score interface, written from the papers in NumPy and PyTorch. Covers neighbourhood, closed-form linear, confidence-weighted MF, pairwise MF, and graph convolution with an optional social graph.

• src/encore/search.py: discovery search. Word-level BM25 retrieval over artist tag documents, 12 features (query match, popularity, three CF scores, user taste vs. artist tags, user interest in the query, friends' listening), and LambdaMART re-ranking.

• src/encore/metrics.py: NDCG / Recall / MRR, coverage, popularity bias, bootstrap CIs, and paired significance tests.

• src/encore/engine.py, api.py, mcp_server.py: the benchmarked model served as is. Every search result can be explained with per-feature TreeSHAP contributions.

• tests/: 20 offline tests on a synthetic catalog, including the leakage guards. CI also smoke-runs both benchmarks on the real dataset.

• docs/METHODOLOGY.md and docs/REFERENCES.md: the task definitions, leakage controls, credits, and limitations, written out in full.

Results

All numbers are produced by the scripts and stored in results/. The full tables, including every paired test and the chosen hyperparameters, are in results/RESULTS.md.

1. Personalized discovery search

A listener types a genre or mood and wants artists new to them. Queries are top tags of each user's held-out artists. Every ranker sees the same BM25 top-200 candidates, which contain 81% of the relevant artists.

Comparing full LambdaMART with non-personalized LambdaMART separates the value of personalization from the value of switching to a learned ranker. Both use the same model class and training data; only the personal features differ.

Personalization matters most where the query alone says least. For broad tags shared by more than 123 artists, non-personalized ranking is close to guessing (NDCG@10 0.04–0.09) and the personal ranker reaches 0.21. The SHAP breakdown shows the same picture. After the query-tag match, the ranker relies on ItemKNN similarity to the user's history, on how well the user's taste matches the artist's tags, and on iALS. The friend-graph features carry almost no weight, and dropping them changes nothing measurable.

The same query from two listeners, served by the trained model (Engine.search):

The personalized list moves toward the electronic and dream-pop side of post-rock, which fits that listener's history.

2. Recommendation

The ranking is a cost/quality decision, not only a leaderboard. LightGCN+Social wins on accuracy and covers 50% of the catalog in top-10 lists, against 29% for EASE. EASE gets 98% of the accuracy in under a second with no tuning beyond one regularizer. That is why the served search stack uses EASE as one of its CF features, alongside iALS and ItemKNN.

What's established vs. what's new here

Every model, metric and protocol in this repo comes from published work, credited in docs/REFERENCES.md. No new algorithm is claimed. What this repo adds is a set of controlled measurements on one public dataset:

New measurements. We did not find these reported elsewhere for HetRec Last.fm.

1. The size of the personalization effect, isolated from the ranker. On identical candidates, with the same learner and training queries, adding personal features lifts discovery-search NDCG@10 by +40.7%. The gain over the classic BM25 + popularity baseline is +110%, or 0.276 NDCG. The controlled comparison splits that gain: about half (0.148) comes from the personal features, and the rest from learning to rank at all.

2. Collaborative filtering still adds value inside search once taste profiles are present. Removing the three CF features (keeping tag-based taste) costs 7.3% (p ≈ 1e-42). Tag-based user profiles alone, the classic folksonomy approach, leave measurable accuracy on the table.

3. The friend graph helps recommendation slightly but adds nothing to search. +1.7% for LightGCN with vs. without friend edges (p = 0.03 on split 0; positive on all three splits, but close to zero on one). Exactly 0 in search (−0.1%, p = 0.94) once CF features are present. In this data, friends' signal appears to be already captured by co-listening.

Results that confirm prior findings (credited, not claimed as discoveries):

• Personalization pays off most on broad queries (5.4× on broad tags vs. 1.7× on narrow ones). This matches Teevan, Dumais & Liebling (SIGIR 2008): personalization helps most where intent varies across users.

• A well-tuned linear model (EASE) gets within 2% of the best graph neural model at about 1% of the training cost. This matches Ferrari Dacrema et al. (RecSys 2019) and Steck (2019).

How the evaluation avoids fooling itself

1. Artist tag profiles drop every tag event on a held-out (user, artist) pair, so a user's own tags cannot leak into queries or features.

2. All search features come from models fit on the training split only.

3. Artists the user already plays are never candidates, so a model cannot win by re-surfacing them.

4. Hyperparameter grids, linear-blend weights, LambdaMART early stopping and neural early stopping all use validation data. Test is scored once per split.

5. Results are reported over three random splits, with 95% bootstrap CIs and paired Wilcoxon tests for every claim above.

Limitations are spelled out in docs/METHODOLOGY.md. The main ones: the dataset is small (1.8K users, 2.8K artists), queries are tags rather than logged search strings, and nothing here was tested online.

Reproduce

git clone https://github.com/Brennan-Richards/encore && cd encore
python -m venv .venv && source .venv/bin/activate
make install        # pip install -e ".[serve,dev]"
make test           # 20 offline tests, ~5 s
make all            # download (2.5 MB, SHA-256 pinned) → recsys (~17 min) → search (~4 min) → report

On an Apple M3 laptop CPU, the full recommendation benchmark (tuning plus three splits) takes about 17 minutes and the search benchmark about 4. make smoke runs both with tiny grids in about a minute.

Serve it

make serve          # http://localhost:8000/docs
curl "localhost:8000/search?q=post-rock&user_id=2&k=5"
curl "localhost:8000/explain?q=post-rock&user_id=2&artist=Sigur%20R%C3%B3s"

To give Claude (or any MCP client) the tools search_music, recommend_artists, similar_artists, user_profile and explain_ranking:

{
  "mcpServers": {
    "encore": { "command": "/path/to/encore/.venv/bin/python", "args": ["-m", "encore.mcp_server"] }
  }
}

The served engine is the split-0 model exactly as benchmarked. User profiles therefore show only training-split artists. Anonymous queries use the tuned BM25 + popularity ranker, because the personal ranker was never trained on users without history.

Next steps

• Natural-language queries: an LLM rewrites "something moody for a rainy drive" into catalog tags, evaluated with an LLM-as-judge against held-out listening.

• Two-tower candidate generation with approximate nearest-neighbour search in place of BM25-only retrieval, and recall@K measured at each stage.

• Navigational queries (artist names) and a re-finding setting, to complement discovery.

• A larger, timestamped dataset (e.g. LFM-1b-style session logs) for a temporal split.

Credits

• Data: Last.fm. The HetRec 2011 Last.fm-2k release was built by Ignacio Fernández-Tobías, Iván Cantador and Alejandro Bellogín (UAM) and is distributed by GroupLens for non-commercial use. It is cited as Cantador, Brusilovsky & Kuflik, HetRec 2011, RecSys 2011. The data is downloaded at run time and not redistributed here.

• Methods: every model, metric and protocol is cited, with the file that implements it, in docs/REFERENCES.md.

• Development: built with Claude Code as an AI pair-programmer. Commits carry a Co-Authored-By trailer.

License

MIT for the code. The dataset keeps its original terms.