Recommender Systems · Machine Learning · Evaluation
Recommendation Engine from Scratch
Five recommenders built from first principles on 568,454 Amazon reviews, compared with ranking metrics, significance tests and a popularity-bias audit.
- Problem
- Recommender comparisons often stop at a single accuracy number. On extremely sparse data, which method actually ranks best — and at what cost to fairness?
- Approach
- Implemented popularity, user-based CF, SVD, TF-IDF content-based and a hybrid ensemble with NumPy and SciPy, then evaluated with leave-one-out HR@10, NDCG@10 and MRR, statistical tests and a bias audit.
Key results
- HR@10 — user-based CF, best of five
- 0.026
- better than random selection
- 12.8×
- effect size behind p = 0.022
- d = 0.18
01
Overview
Five recommendation systems implemented from first principles on the Amazon Fine Food Reviews dataset (568,454 reviews), then compared with a ranking-focused evaluation framework, statistical testing and a fairness audit.
02
Problem & motivation
Library recommenders hide the decisions that matter. Building each approach by hand made it possible to see why one wins on sparse data — and to measure what each one costs in popularity bias.
03
My role
- Independent project — implementation, evaluation design, statistical analysis and write-up.
04
System architecture
Stage 1:Data
Pipeline
EDA, k-core filtering, sparse user × item matrix (0.195% dense)
Stage 2:Recommenders
Popularity baseline
Bayesian average
User-based CF
Cosine similarity, k = 30
SVD
Sparse matrix factorization, k = 50
Content-based
TF-IDF, 8K features + bigrams
Hybrid ensemble
Weighted blend of all four
Stage 3:Evaluation
Ranking
HR@K, NDCG, MRR — leave-one-out, 500 users
Rating error
RMSE / MAE
Stage 4:Analysis
Significance
t-test, Mann-Whitney U and Cohen's d
Bias audit
Four-tier popularity exposure framework
05
Technical implementation
Neighborhoods vs. latent factors
User-based CF uses cosine similarity over the 30 nearest neighbors; SVD factorizes the sparse matrix with SciPy's svds at 50 latent factors. Both run on the same k-core-filtered matrix.
Content signal
A TF-IDF model over review text (8K features with bigrams) gives a recommender that doesn't depend on co-ratings at all.
A hybrid, measured honestly
The hybrid blends SVD (0.40), CF (0.25), content (0.20) and popularity (0.15). It was evaluated like every other system rather than assumed to be best.
06
Engineering decisions & challenges
User-CF beats SVD on very sparse data
At 0.195% density, neighborhood signal is stronger than weak latent factors. Algorithm sophistication is not a substitute for data density.
The hybrid is not automatically better
Giving SVD the highest weight, when it is the weakest personalized system, pulls the hybrid away from its strongest signal (CF).
Combining systems amplifies bias
The hybrid's exposure to the most popular items (56.3%) exceeds every individual system — three signals independently favor popular products.
Effect size with every p-value
p = 0.022 with Cohen's d = 0.18 means the difference is statistically real but practically small.
07
Evaluation & results
Leave-one-out ranking evaluation over 500 users. Absolute hit rates are low because the matrix is extremely sparse — the comparison between systems is the point.
- HR@10, user-based CFBest of the ranked systems
- 0.026
- better than random selection
- 12.8×
- bias score, user-based CFSecond-fairest system (lower is better)
- 110.3
Ranking quality and popularity bias by system
User-based CF leads every ranking metric and is the second-fairest system; the hybrid concentrates exposure on popular items.
- User-based CF0.026
- Hybrid0.018
- SVD0.012
- Popularity0.002
Reference line: Random at 0.002.
Show data tableHide data table
| Item | Value |
|---|---|
| User-based CF | 0.026 |
| Hybrid | 0.018 |
| SVD | 0.012 |
| Popularity | 0.002 |
- User-based CF0.0141
- Hybrid0.0095
- SVD0.0060
- Popularity0.0006
Show data tableHide data table
| Item | Value |
|---|---|
| User-based CF | 0.0141 |
| Hybrid | 0.0095 |
| SVD | 0.0060 |
| Popularity | 0.0006 |
- User-based CF0.0106
- Hybrid0.0069
- SVD0.0042
- Popularity0.0002
Show data tableHide data table
| Item | Value |
|---|---|
| User-based CF | 0.0106 |
| Hybrid | 0.0069 |
| SVD | 0.0042 |
| Popularity | 0.0002 |
- User-based CF110.3
- Hybrid145.9
- SVD100.8
- Popularity130.2
Show data tableHide data table
| Item | Value |
|---|---|
| User-based CF | 110.3 |
| Hybrid | 145.9 |
| SVD | 100.8 |
| Popularity | 130.2 |
Source: Repository README results table (leave-one-out, 500 users)
| System | HR@10 | NDCG@10 | MRR | Bias ↓ |
|---|---|---|---|---|
| Popularity baseline | 0.0020 | 0.0006 | 0.0002 | 130.2 |
| SVD | 0.0120 | 0.0060 | 0.0042 | 100.8 |
| Hybrid ensemble | 0.0180 | 0.0095 | 0.0069 | 145.9 |
| User-based CF | 0.0260 | 0.0141 | 0.0106 | 110.3 |
- Dataset: Amazon Fine Food Reviews (SNAP). Results are offline evaluation, not online A/B tests with live users.
08
From the notebook
Real figures from the project's notebook, as published in the repository. Select one to enlarge it.
09
Lessons learned
Data density beats model sophistication
The simplest personalized method won because the data was sparse — a reminder to diagnose the data before reaching for a bigger model.
Accuracy and fairness can coexist
User-based CF was both the most accurate and the second-fairest system; the trade-off isn't always a trade-off.
10
Technologies used
- Python
- NumPy
- SciPy
- scikit-learn
- pandas
- Matplotlib
Want the deeper walkthrough?
I'm happy to go through the code, the trade-offs and what I'd do next.