Movie Ratings

The full MovieLens dataset, which contains 27M ratings given to 58K movies by 280K users.

Movie Metadata

Data retrieved from the TMDB Open API and available as a Kaggle Dataset. It contains information on 45k movies featured in the full MovieLens dataset. Features include title, description, genre and cast&crew information.

Movie Embeddings and Similarity Matching

First of all, we use both interaction and content data to extract item embedding\(^5\) features, which are low-dimensional dense vectors that captures some of the semantics of the item by placing semantically similar items close together in the embedding space. It allows efficient nearest neighbors search, so that we can find similar movies fast in application time.

• Content Embedding: embedding features extracted from text fields like title and description, and from categorical fields like genre and cast&crew.
• Collaborative Filtering Embedding: embedding features extracted from user-movie interactions given by the ratings.

All the details about the feature extraction process will be discussed later.

After computing the embedding features, similar movies will be recommended using Approximate Nearest Neighbors (ANN) search with cosine similarity. The cosine similarity is the most common affinity metric applied for similarity search in the embedding space.

Users Intersection as a Similarity Measure

Let’s consider every rating (from 1 to 5) above 3.5 as an explicit feedback that the user liked the movie. Then we can measure the similarity between a pair of movies \((x,y)\) as:

\[ S(x,y) = \displaystyle\frac{N_{xy}}{\sqrt{N_xN_y}} \]

where \(N_x\) is the number of users that liked movie x, \(N_y\) is the number of users that liked movie y, and \(N_{xy}\) is the number of users that liked both x and y.

So, \(S(x,y)\) close to 1 is a strong evidence that users who liked x will like y too. We call this as User Correlation.

Despite providing good recommendations, this similarity is very sparse, resulting in a discrete recommendation. So, if we try to recommend the top 30 similar to each movie we will get less than 30 for many movies. In other words, we will have many movies with few or none similar items to be recommended.

The embedding features solve this problem. Since we are computing distances between pairs of movies in a dense and low-dimensional space, the embedding features provide a continuous recommendation. We are able to find similar items to any movie, with high evidence (low distance) or low evidence (high distance).

Evaluating the Embedding Features

How close the similarity match with embedding features can get to the similarity match given by the user’s likes (user correlation)?

(a) Ranking correlation

In a attempt to answer this question, we can randomly select \(n\) test movies and get their top N similar using the \(S(x,y)\) measure. Then, for each test movie, we get the distances \(D(x,y)\) from the top N similar using the embedding features. Finally, for each test movie, we apply the Spearman’s Rank-Order Correlation Coefficient (\(\rho\)) between \(S\) and \(1/D\), which is equal to the Pearson’s correlation between the rank values of them.

For all test movies, we get an average rank correlation:

\[ \frac{1}{n}\sum_{i=1}^n\rho(S_i, 1/D_i) \]

(b) Precision and Recall at K

Given a threshold \(t_S\) for \(S(x,y)\), we will assume that \(S(x,y) \geq t_S\) indicates a strong/relevant similarity. So, given the K nearest neighbors from embedding features of a test movie:

\[ \text{precision@k} = \frac{\#\text{relevants@k}}{K} \]

\[ \text{recall@k} = \frac{\#\text{relevants@k}}{\#\text{relevants}} \]

For all test movies, we get an average \(\text{Precision@K}\) and an average \(\text{Recall@K}\).

Ratings Distribution

First of all, let’s observe the distribution of ratings. We have 10 possible values for the user rating: \([0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0]\). In which proportion each one of these values appear in the data?

Just a small part of the ratings fall bellow 3.0. If we look at the movie average rating we will observe a similar behavior:

In general, users tend to give good ratings to the movies they vote. It may be an evidence that users are most likely to vote in those movies they like. It makes sense, since users usually watch movies with actors and/or directors they already know and like, and in the genre they like most.

Ratings as Likes and Dislikes

In this project we will convert ratings to likes (>=3.5) and dislikes (<3.5). Let’s take a look the the distribution of number of likes:

We got a very asymmetric empirical distribution, so we applied the log transformation to make it easy to visualize the full distribution. Just to have an idea, 10 in the log scale corresponds to about 22025 likes.

Now, let’s see how the number of dislikes behave:

Is there any movie with more dislikes them likes?

Since we have negative values, the answer to the last question is yes.

Is there a consistent bias in the average rating to popular movies?

Yes, we can see a popularity bias. Movies with more than 20k votes have average rating above 3.0. Movies with more than 50k votes tend to have average rating above 4.0.

Most Liked and Most Hated Movies

Which movies are the most liked by the users?

How about the most disliked movies by the users?

Genre Distribution

Now, let’s check the distribution of movie genres:

The Drama genre is present in about 20% of the movies, it’s very frequent. On the other hand, now I understand why it’s so hard to find new Western movies to watch. Just 1.5% of the movies are Western.

Most Liked Western and Horror Movies

Let’s take a look at my personal favorite genres, Western and Horror. Which movies are the most liked by the audience?

Western Movies:

Horror Movies:

I totally agree with the ranking!

Most Liked and Most Hated Directors

Can we find the public opinion about the movie directors based on the number of likes and dislikes given to the movies they worked on? To answer this question, we subtracted the number of likes by the number of dislikes considering all movies each director worked. Here it follows the ranking of the most liked and most disliked directors:

The Schindler’s List list is in the top 10 most liked movies and helped Steven Spielberg to be the first in the ranking.

On the other hand, Ace Ventura, the first in the list of the most disliked movies, put two directors in the list of the 10 most disliked ones.

Most Liked and Most Hated Actors

How about the actors?

The The Shawshank Redemption put two actors in the list of the 10 most liked actors!

The Ace Ventura definitely didn’t fall in popular taste.

Building Content Embeddings

For text features (title, description, keywords and user comments/tags) we concatenate them all and extract a sparse TF-IDF matrix using the TfidfVectorizer.

For the categorical features (genre, cast and crew) we expand them into sparse binary features.

Both representations result in high-dimensional and sparse matrices. So, we combine them and apply the sparse TruncatedSVD algorithm to get low-dimensional and dense representations, the embedding features.

The implicit (link) package provides some fast factorization algorithms more powerful than SVD. They are great alternatives and may improve the representations.

Building CF Embeddings

The surprise (link) package provides many classical factorization algorithms for explicit feedback problems. We can apply than in the user-movie ratings matrix to extract latent factors for the users and movies. Then, the movie factors can be used as embedding features.

However, we decided to try an approach based on likes and dislikes. Ratings above 3.5 will be considered likes (+1), while ratings below 3.5 will be considered dislikes (-1). From this assumption, we can build a user-movie matrix with (-1,+1) values and use a logistic loss to factorize this matrix. The LighFM (link) package provides a logistic loss factorization algorithm with additive bias. So, it takes into account the user and the movie bias, abstracting popularity away from the embedding features. This is an attempt to solve the item popularity problem.

Similarity Match

After computing the embedding features we recommend similar movies using an approximate nearest neighbors search with cosine similarity. The faiss package is our choice, since it provides an extremely fast and scalable ANN algorithm.

Hybrid Approach

First, we evaluate the recommendations given by the content embedding features and compare the result with the performance for the CF embedding features. Then, as a final attempt, we test a hybrid approach which works as follows:

1. Select a target movie for which we will recommend similar ones.
2. Compute the cosine similarity between the target and the candidates using the content embedding features.
3. Compute the cosine similarity between the target and the candidates using the CF embedding features.
4. Order the candidates by the sum between the computed similarities in the last 2 steps.

It’s the same as combining the embedding features (content based + collaborative filtering) and taking the cosine similarities over them all. Since this approach combines content information with interaction information, it may provide better recommendations.

Text Features

First of all, we combined all text fields (title,overview,tagline, tags and keywords) into a single one. Then we applied some text cleaning:

• Lower case.
• Remove numbers, punctuation and special chars.
• Remove single char words.
• Remove stopwords.

To encode text into numerical features we used the TfidfVectorizer method with the following parameters:

• Uni-grams and bi-grams.
• Vocabulary of 30k terms.
• Minimum document frequency of 5.
• Maximum document frequency of 50%.
• Scale rows by their l2-norm.

We got a Tf-Idf matrix with dimension 20624 x 30000 and 99.79% of sparsity.

Since the cosine similarity between two vectors is their dot product when their l2-norm is equal to 1, we always apply this normalization. The faiss package requires normalized vectors to find nearest neighbors.

All code is available in the 04-content-based-embedding.ipynb notebook.

Categorical Features

Genre and Cast&Crew are categorical features. To improve content representation, we need to combine them with the Tf-Idf features. We may try a classical one-hot-encoding process, but these features have high cardinality. Representing them by a dense binary matrix may consume lots of RAM and make the embedding extraction very slow. So we represent them using sparse matrices and apply l2-normlization on rows.

For Crew we got a 20624 x 3635 sparse matrix with 99.81% of sparsity. It means that we are consuming just 0.19% of the memory required by a dense matrix. For Cast we got a 20624 x 10269 sparse matrix with 99.94% of sparsity. Finally, after combining the binary features with the Tf-Idf ones we got a 20624 x 43904 sparse matrix with 99.84% of sparsity.

So, we have a high dimensional and high sparse content matrix to work with.

We decided to not include genre in the content matrix. We already have many information on text features about the movie, more specific than genre. Genre gives just a few binary features and may bring redundant information.

All code is available in the 04-content-based-embedding.ipynb notebook.

Ratings

For the ratings dataset we need to build an Utility Matrix. In other words, we need to put data into an user/movie matrix format. Users will be represented as rows, movies as columns and like(+1)/dislike(-1) as values. Since we have a large number of movies and the greatest part of the user/movie interactions are missing (a user votes in a small part of the movies, and a movie is evaluated by a small part of the users), a sparse representation is the better choice.

We transformed ratings into likes (rating >=3.5 to 1.0) and dislikes ( rating <3.5 to -1.0) and got a 273627 x 10698 sparse matrix with 99.13% of sparsity.

All code is available in the 05-collaborative-filtering-embedding.ipynb notebook.

Content Based

We extract dense features from the sparse content matrix using the TruncatedSVD algorithm. The parameter algorithm="arpack" is necessary to deal with a sparse matrix. It took less than 20 seconds to factorize the content matrix.

We use 300 as the embedding dimension. It’s a very common value in Word Embedding representations.

Let’s take advantage of the Embedding Projector tool to visualize the embedding features:

The Lord of the Rings.

Star Wars.

Pulp Fiction.

The nearest neighbors shown make sense.

For more details, check the 04-content-based-embedding.ipynb notebook.

Collaborative Filtering

We extract dense features from the utility matrix using the LightFM algorithm with logistic loss (to deal with +1/-1 values). It took about 4 minutes to extract the user and item factors. The last are our embedding features. As in the content based approach, we set 300 as the embedding space dimension.

The LightFM algorithm learns the user and item factors with a bias term. It’s important to overcome the popularity bias problem. Let’s check how it relates to the movie average rating:

Item bias fitted by the LastFM algorithm against the movie average rating.

We see that the bias term captures the movie rating information. This way we remove the popularity bias from the embedding features.

Now, let’s visualize the embedding features using the Embedding Projector:

The Lord of the Rings.

Star Wars.

Pulp Fiction.

We have different neighbors when compared to the content based, but they make sense too.

Hybrid Approach

It consists on combining content based with collaborative filtering embedding features. Computing the cosine similarity over this combination is the same as computing the cosine similarity individually over contend based and collaborative filtering and taking the average between the two values.

All code is organized and documented in the 08-hybrid-approach.ipynb notebook. In the evaluation section we will show that content based and collaborative filtering are complementary solutions, which may be a evidence that the hybrid approach is more powerful.

Similarity Match with ANN

Using the faiss package we can efficiently find similar movies based on the cosine similarity.

In the 06-similarity-match-with-ann.ipynb notebook we experimented with the embedding features using the faiss package. First we got the 30 nearest neighbors to each movie, which we will use to compute precision and recall at k. Then, we computed the cosine similarity between each movie and its 30 nearest neighbors given by user correlation. They will be used to compute ranking correlation in the next section.

Public Movie Embedding

We published the embedding features (content based + collaborative filtering) on the Embedding Projector. You can play with it here: Movie Embedding.

More details can be seen in the projector folder.

Movie Similarity Dashboard

Finally, we built a web application on which the user can choose a movie and get recommendations of similar movies, using the Streamlit (link) framework:

.

We provide a Dockerfile to easily deploy the application in your computer. Follow the instructions in the project README.md.

Ranking Correlation

In the Similarity Match section we have computed the cosine similarity between pairs of movies. Now, we compare the user correlation ranking with the cosine similarity ranking based on the embedding features, using the Spearman correlation.

The Spearman Ranking Correlation Coefficient give us a value between -1 (exactly opposite order) and +1 (same order). Like the classical Pearson’s Correlation Coefficient, values above 0.3 can be seen as an evidence of correlation.

For each target movie (the 500 movies selected in the Benchmark section) we computed the Spearman correlation between the user similarity values and the cosine similarity values for the movie’s 30 nearest neighbors. To evaluate the performance, we can observe the distribution of the correlation measures for all 500 target movies, as follows:

.

The embedding features based on text data performed a little bit better than other approaches. It’s a surprise, since text data does not carry any information about users. But there is a large overlapping between the distributions. So, there is no strong evidence of difference between methods.

In addition, including Cast&Crew information doesn’t seem to improve the performance of the content embedding features. The Pearson’s correlation between the Tf-Idf results and the Tf-Idf + Cast&Crew results is above 0.7. So, Cast&Crew seems to provide redundant information to the embedding features.

Precision and Recall

Another way to check ranking performance is to transform the problem into a classical binary classification problem, so that we can take advantage of better known metrics. Precision and recall are the most common ones.

First of all, we must label some items as ‘relevant’. Than, we look for relevant items in the top K positions of the ranking. K in [5,10] is a common choice. Here we choose k=10.

Precision is given by the number of relevant items in the top K divided by K. In other words, it measures the proportion of the top K items that are relevant. Recall is given by the number of relevant items in the top K divided by the total number of relevant items. In other words, it measures the proportion of relevant items that are in the top K items.

We assigned as relevant all similar movies with user correlation above 0.3. Then, to make a better evaluation, we considered only those target movies with 3 or more relevant similar movies.

We got very bad results for precision and recall. In general, the relevant movies are not appearing in the top 10 given by the embedding ranking. Here it follows what we got:

• Precision for the Tf-Idf Embedding:

• Recall for the Tf-Idf Embedding

• Precision for the CF Embedding

• Recall for the CF Embedding

Popularity Bias

In addition, we checked if the embedding features carry any type of popularity bias.

As expected, content based embedding is not subjected to popularity bias:

Movie popularity against ranking correlation values for Content Embedding

How about the collaborative filtering embedding?

Movie popularity against ranking correlation values for Collaborative Filtering

No popularity bias can be observed, indicating that the bias term in factorization algorithm was fitted properly.

Correlation between Content Based and Collaborative Filtering Embedding

We saw that Cast&Crew brings redundant information to the Content Embedding. How about Collaborative Filtering? Can we expect similar results between Content Based and Collaborative Filtering Embedding?

Ranking correlation results for Content Based Embedding against Collaborative Filtering Embedding

We can observe that there is no strong correlation between text embedding and collaborative filtering embedding. The Pearson’s correlation between than is 0.2. It may be a indicative that both methods are complementary. So, combining than can provide a even better performance. It is the intuition behind the Hybrid Approach.