Recommendation and ranking systems
A recommender decides what to show you next. It works by turning people and items into lists of numbers and finding the pairs that fit.
Words on this page (6)
- recommender
- : A system that suggests items such as videos or products to each user.
- collaborative filtering
- : Recommending items by finding people with similar past choices: if they liked it, you probably will too.
- embedding
- : A list of numbers that represents the meaning of a piece of text or an image, so similar things sit close together.
- dot product
- : Multiply matching entries of two vectors and add them up. Large when they point the same way.
- vector
- : An ordered list of numbers. Embeddings and model activations are vectors.
- cold start
- : The problem that a new user or item has no history, so a recommender has nothing to match.
What it is
A (or recommendation system) suggests items: videos, songs, products, posts. The classic idea is : people who liked the same things in the past will probably like the same things in the future. You do not need to know anything about the items, only who interacted with what.
Modern systems learn an (a list of numbers) for each user and each item, so a good match is a high , the same operation you meet in the Building Blocks chapter. Large services work in two steps. First a fast stage picks a few hundred candidates from millions. Then a slower, more careful model ranks those few hundred.
A worked example
A user is the [0.9, 0.1]: loves action, barely cares about romance. Movie A is [0.8, 0.2], mostly action. Movie B is [0.1, 0.9], mostly romance. Dot product with A: 0.9 × 0.8 + 0.1 × 0.2 = 0.74. With B: 0.9 × 0.1 + 0.1 × 0.9 = 0.18. A ranks first. In real systems the vectors have hundreds of numbers and the model learns them from clicks and watch time.
What goes wrong
The problem: a new user or a new item has no history, so there is nothing to match. Systems fall back to popular items or ask a few questions. A feedback loop: if you only show what people already click, you never learn what else they would like. The target matters too: optimizing clicks can promote sensational items, while optimizing "watched to the end" rewards something different.
How collaborative filtering finds a match
Three users and four movies. Ana liked movies 1 and 2. Ben liked 1, 2 and 3. Cy liked only 4. Ben shares both of Ana's likes, Cy shares none. So Ben is Ana's nearest neighbor, and the system suggests movie 3 to Ana, which Ben liked and she has not seen. Embeddings do the same job with numbers and scale to millions of users.
Why two stages
Invented numbers: a careful model takes 2 milliseconds per item. Scoring 5,000,000 items takes 10,000 seconds, nearly three hours, for one page view. The cheap first stage cuts that to 500 candidates, and the careful model then scores them in 1 second. The cost is that anything the first stage drops can never be recommended, so a weak first stage caps quality.
What goes wrong: counts are not rates
Two videos are equally liked: half of the viewers click either one. The system shows video X to 900 users and video Y to 100. X gets 450 clicks, Y gets 50. Ranked by click count, X looks nine times better, and gets shown even more tomorrow. Compare clicks per view instead, and both are 50 percent. To confirm, check whether the top items are simply the ones shown most. Showing a small share of random items keeps feeding the system fresh evidence.
How to tell a new ranking is better
Split 20,000 users in two equal groups. Version A gets a click rate of 4.0 percent and version B 4.3 percent. The random wobble in the difference between two groups of 10,000 is about 0.28 percentage points. The measured gap is 0.3, barely more than one wobble, so it could be luck. You would need more users or a longer test before shipping B.
Common mistake
Measuring only accuracy on past data. What counts is what happens when real users see the new ranking, so teams run A/B tests that compare two versions on live traffic.
In 2006 Netflix offered one million dollars to anyone who could improve its movie predictions by 10 percent. A team won in 2009 with a blend of many models. Netflix later said it never ran the full winner, because the engineering cost was too high for the gain. Simple and good enough often beats best on paper.
Check yourself
What does collaborative filtering rely on?
Not quite. It only needs the record of who interacted with which items.
A brand new user opens the app. Which problem appears?
Not quite. With no past behavior, the system has nothing to compare.
Why do big systems use a candidate stage first?
Not quite. A fast filter narrows millions to hundreds so a careful model can rank them.
You got 0 of 3 right.
Go deeper
- CourseRecommendation SystemsCourse
Google · A short course on candidate generation, scoring and ranking.
- ArticleNetflix PrizeArticle
Wikipedia · The contest that made collaborative filtering famous, and why the winning model was not shipped.
- Article
Eugene Yan · How real recommenders split into candidate retrieval and ranking.
Spotted a mistake or something unclear? Report it on GitHub.