Genres Where Audiences and Critics Disagree Most

This article was created with Jupyter. To see the code generating all figures and outputs, click 'Show Code'

Introduction¶

I recently browsing the Rotten Tomatoes page all-time favorite sci-fi series, Firefly. I was surprised that critics were not impressed with the series (average rating of 7.96/10). Among 'Top Critics', this score dropped to an average of 7.25/10. But everyone I know who has seen it loves the series!

Unwilling to accept that my beloved sci-fi franchise deserved this low rating from critics, I thought that maybe this was a bias with science fiction. Critics end up seeing most big movies and shows regardless of their personal tastes, so on average they might not be geeky enough to really appreciate a good science fiction movie. The typical audience who has decided to view science fiction, on the other hand, presumably are geeks who really 'get' science fiction. So we might expect audiences to rate good science fiction higher than critics, because the geeks who see science fiction really 'get it'. Consistent with this, the average audience rating of Firefly is 9.2/10 - a score closer to what I (a self-styled science-fiction connoisseur) think it really deserves.

Is this true of the genre more generally? Do audiences tend to appreciate (or at least more highly rate) science fiction media more than the critics? Unsatisfied with just hypothesizing about this, I decided to look at some data. It turns out I was very wrong.

Data Source and Disclaimers¶

The data I analyzed is from the 10 most popular movies every year from 1975-2015. I obtained the data from Crowdflower. The data used was largely a matter of convenience - it was the first data source I found that was fairly large and had genre labels, audience ratings, and critic ratings. That said, since these are 'blockbuster movies', the results might be a bit different than if we included less popular movies.

In [1]:

%matplotlib inline

#Import some tools we'll use
import pandas as pd
import numpy as np
from bokeh.palettes import Category20, Category20b
from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, Range1d, LabelSet, Label, HoverTool
from bokeh.charts.attributes import CatAttr
from bokeh.charts import Bar
from scipy.stats.mstats import zscore

output_notebook(hide_banner=True)

#Display floats in pandas tables only to one decimal
pd.options.display.float_format = '{:,.1f}'.format

In [2]:

import bokeh 
bokeh.__version__

Out[2]:

'0.13.0'

In [2]:

##DATA PREP

#Read in the data
mov = pd.read_csv('blockbuster-top_ten_movies_per_year_DFE.csv')

#Remove columns where data is missing
mov = mov[(mov['rt_score'] != 0) & (mov['rt_audience_score'] != 0) & (~pd.isnull(mov['Genre_1']))]

#rt_score is on a 1 to 10 scale, rt_audience_score is 0.5 to 5. Let's rescale
mov['rt_audience_score'] = mov['rt_audience_score']*2

#Only include those columns we want
mov = mov[['title', 'rt_audience_score','rt_score','Genre_1','Genre_2','Genre_3']]

#Calculate difference between audience and critic scores (to use later)
mov['dif'] = mov['rt_audience_score'] - mov['rt_score']

#Generate a unique color based on whatever the first genre is for each movie, for plotting
cat = Category20[20]
cat.append(Category20b[3][0])
genres = mov['Genre_1'].unique()
mov['colors'] = [cat[np.where(genres == genre)[0][0]] for genre in mov['Genre_1']]

#Generate a genre string based on the three genres (to prevent hover tooltips from displaying nans)
mov['Genre_str'] = (mov['Genre_1'] + ' ' +
                    mov['Genre_2'].fillna('') + ' ' +
                    mov['Genre_3'].fillna(''))

In [3]:

#Themes aren't really supported yet in Bokeh, so define a function to do styling
def styleBokeh(p):
    #Hide the bokeh toolbar
    p.toolbar_location = None
    
    #Format axis label fonts
    p.yaxis.axis_label_text_font_size = "12pt"
    p.yaxis.axis_label_text_font_style = "normal"
    p.xaxis.axis_label_text_font_size = "12pt"
    p.xaxis.axis_label_text_font_style = "normal"
    
    p.xaxis.major_label_text_font_size = "10pt"
    p.yaxis.major_label_text_font_size = "10pt"
            
    #Change title font size
    p.title.text_font_size='14pt'

Critics Are More Critical?¶

Here is a plot of each movie's audience and critic ratings, coloring by one of their genres. You can hover over a point to find out what movie it corresponds to. Note that most movies have multiple genres, so the coloring is a bit arbitrary. The red diagonal indicates where audience scores and critic scores are equal. Above the line means audiences rate the film higher, below means critics rate it higher.

In [4]:

##SCATTER PLOT ALL MOVIES

#Make a data source for bokeh to use
mov_source = ColumnDataSource(mov)

#We want the axes on the same scale, so figure out min of both and max of both
low = mov[['rt_audience_score','rt_score']].min().min() - 0.2
high = mov[['rt_audience_score','rt_score']].max().max() + 0.2

#Define the hover over tooltips
hover = HoverTool(
        tooltips=[("Title", "@title"),
                  ("Audience Score", "@rt_audience_score{0.0}"),
                  ("Critic Score", "@rt_score{0.0}"),
                  ("Genres", "@Genre_str")
        ]
    )

#Make the figure
p = figure(title = "Movie critic and audience ratings",
           x_range=Range1d(low, high),
           y_range=Range1d(low,high),
           tools=[hover])

#We'll put critic ratings on the x-axis and audience on the y-axis
p.xaxis[0].axis_label = 'Critic Rating'
p.yaxis[0].axis_label = 'Audience Rating'

#Add a diagonal line for where audience rating = critic rating
p.line((low,high), (low,high),line_color="red", line_width=5, alpha=0.5)

#Plot the points, coloring by Genre_1
p.scatter(x='rt_score', y='rt_audience_score', source=mov_source, color='colors', 
          size=8, fill_alpha = 1)

styleBokeh(p)
show(p)

One immediately obvious observation is that critics just use a wider range of ratings. Specifically, critic scores go much lower than audience scores. Average critic scores go as low as 2.7/10 (for The Blue Lagoon and Staying Alive) and as high as 9.1 (for E.T., Raiders of the Lost Ark, and All the President's Men). Audience scores, on the other hand, only go from 5 (Godzilla, The Flinstones) to 8.8 (The Avengers, The Dark Knight, Guardians of the Galaxy). Unsurprisingly, the average critic score is lower than the average audience score - 6.58 for critics compared to 6.96 for audiences. This bias for critics to rate films lower than audiences is consistent with what others have found.

So across the board, audiences rate films higher than critics. But does science fiction stand out as a genre, taking into account the overall bias? While the points are colored by genre, there aren't any obvious patterns. The data becomes more interpretable when we average across genre.

In [5]:

##DATA PREP FOR GENRE PLOTS

#Melt the genre columns so we only have one. This will give multiple entries for movies with more than one genre
melted_mov = pd.melt(
    mov.drop(['colors', 'Genre_str'],axis=1), 
    id_vars=['title',
             'rt_audience_score',
             'rt_score',
             'dif'],
    value_name='genre')

#A little bit of clean up
melted_mov = melted_mov.drop('variable',1)
melted_mov = melted_mov.dropna()

#Group by genre
genre_groups = melted_mov.groupby('genre')

#Excluding genres with less than 20 entries
genre_revs = genre_groups.mean()[genre_groups['genre'].count() > 20]

In [6]:

##SCATTER PLOT FOR GENRE SCORES

#Make a data source for bokeh
genre_revs_source = ColumnDataSource(genre_revs)

#Get axes ranges
low = genre_revs[['rt_audience_score','rt_score']].min().min() - 0.05
high = genre_revs[['rt_audience_score','rt_score']].max().max() + 0.05

#Define the hover over tooltisp
hover = HoverTool(
        tooltips=[("Genre", "@genre"),
                  ("Audience Score", "@rt_audience_score{0.0}"),
                  ("Critic Score", "@rt_score{0.0}"),
                  ("Difference", "@dif")
        ]
    )

p = figure(title='Audience and critic ratings by genre',
           x_range=Range1d(low, high),
           y_range=Range1d(low,high),
           tools=[hover])

p.xaxis[0].axis_label = 'Critic Rating'
p.yaxis[0].axis_label = 'Audience Rating'

#Add a line for where critic score = audience score
p.line((low,high), (low,high),line_color="red", line_width=5, alpha=0.5)

#Scatter plot the genre scores
p.scatter(x='rt_score', y='rt_audience_score', source=genre_revs_source,
          size=6)

#Add labels for each genre
labels = LabelSet(x='rt_score', y='rt_audience_score', text='genre', source=genre_revs_source, 
              x_offset=0, y_offset=5, text_font_size="7.5pt", text_align='center')
p.add_layout(labels)

#Hide the bokeh toolbar
p.toolbar_location = None

styleBokeh(p)
show(p)

We can see that the trend of audiences liking movies more than critics is consistent across all genres. Interestingly, both audiences and critics seem to love animation films. It's easier to see where audiences and critics disagree most by looking at the difference in their ratings.

In [7]:

genre_revs = genre_revs.sort_values('dif', ascending = False)
p = Bar(genre_revs.reset_index(),
        values='dif',
        label=CatAttr(columns=['genre'], sort=False),
        legend=False,
        title='Difference between critics and audience ratings by genre',
        ylabel='Audience ratings minus critic ratings',
        tools=[])
p.toolbar_location = None
styleBokeh(p)
show(p)

Now we can see my hypothesis was actually quite wrong. Sci-fi falls right in the middle of the pack in terms of how much the audience ratings differ from critics. Instead, fantasy leads the pack in genres that audiences love more than critics.

In [8]:

#Make mov presentable for printing
print_mov = mov.drop(['Genre_1', 'Genre_2', 'Genre_3', 'colors'], axis=1)
print_mov.columns = ['Title', 'Audience Score', 'Critic Score', 'Difference', 'Genres']

Top 10 Fantasy Movies Audiences Love More Than Critics ¶

To get a sense of what kinds of movies are driving the disagreement over fantasy movies, we can look at the movies in this genre where the scores disagree most.

In [9]:

#Get the titles of the 10 fantasy movies with the greatest difference between audience and critics
fantasy_movie_titles = (melted_mov[melted_mov['genre'] == 'Fantasy'].
                        sort_values(by='dif', ascending=False)
                        [0:10]
                        ['title'])

#Make the table pretty and output it
fantasy_table = (print_mov
                 [print_mov['Title'].isin(fantasy_movie_titles)].
                 sort_values('Difference', ascending=False)
)
fantasy_table.index = range(1,11)
fantasy_table

Out[9]:

	Title	Audience Score	Critic Score	Difference	Genres
1	The Twilight Saga: New Moon	7.2	4.7	2.5	Fantasy Drama Adventure
2	The Twilight Saga: Breaking Dawn - Part 2	7.6	5.2	2.4	Fantasy Drama Adventure
3	300	8.2	6.1	2.1	War Fantasy Action
4	Pirates of the Caribbean: At World's End	7.6	5.5	2.1	Fantasy Adventure Action
5	Pirates of the Caribbean: On Stranger Tides	7.0	5.1	1.9	Fantasy Adventure Action
6	The Twilight Saga: Eclipse	7.2	5.4	1.8	Fantasy Drama Adventure
7	Hancock	7.2	5.4	1.8	Fantasy Action
8	Pirates of the Caribbean: Dead Man's Chest	7.8	6.0	1.8	Fantasy Adventure Action
9	Man of Steel	7.8	6.2	1.6	Fantasy Adventure Action
10	Thor: The Dark World	7.8	6.2	1.6	Fantasy Adventure Action

Perhaps obvious in retrospect, The Twilight Saga and superhero movies are where critics and the audience disagree most.

Top 10 Movies Audiences Love More Than Critics¶

We can also look across all movies, what movies do the audiences love much more than critics.

In [10]:

top10_table = print_mov.sort_values('Difference', ascending=False)[0:10]
top10_table.index = range(1,11)
top10_table

Out[10]:

	Title	Audience Score	Critic Score	Difference	Genres
1	Bad Boys II	7.2	4.1	3.1	Crime Comedy Action
2	Transformers: Revenge of the Fallen	7.0	3.9	3.1	Sci-Fi Adventure Action
3	The Blue Lagoon	5.8	2.7	3.1	Romance Drama Adventure
4	Staying Alive	5.4	2.7	2.7	Romance Music Drama
5	Transformers: Age of Extinction	6.6	3.9	2.7	Sci-Fi Adventure Action
6	The Twilight Saga: New Moon	7.2	4.7	2.5	Fantasy Drama Adventure
7	National Treasure: Book of Secrets	7.2	4.8	2.4	Mystery Adventure Action
8	Home Alone 2: Lost in New York	6.0	3.6	2.4	Family Comedy Adventure
9	The Twilight Saga: Breaking Dawn - Part 2	7.6	5.2	2.4	Fantasy Drama Adventure
10	Transformers	8.2	5.8	2.4	Sci-Fi Adventure Action

Movies Critics Love More Than Audiences ¶

What about the reverse? What are the top 10 movies critics rate higher than audiences?

In [11]:

bot10_table = print_mov.sort_values('Difference', ascending=True)[0:10]
bot10_table.index = range(1,11)
bot10_table

Out[11]:

	Title	Audience Score	Critic Score	Difference	Genres
1	King Kong	5.6	7.7	-2.1	Drama Adventure Action
2	E.T. the Extra-Terrestrial	7.0	9.1	-2.1	Sci-Fi Family Western
3	Rocky	6.4	8.3	-1.9	Sport Drama
4	Toy Story	7.4	9.0	-1.6	Comedy Animation Adventure
5	Toy Story 2	7.0	8.6	-1.6	Comedy Animation Adventure
6	Shakespeare in Love	6.8	8.3	-1.5	Romance Drama Comedy
7	The Incredibles	6.8	8.3	-1.5	Animation Adventure Action
8	The Terminator	7.2	8.7	-1.5	Sci-Fi Action
9	Titanic	6.6	8.0	-1.4	Romance Drama
10	Close Encounters of the Third Kind	7.4	8.8	-1.4	Sci-Fi Drama

Again, I have to say, I think I like the critics' tastes better than the audience (though if this data set contained La La Land, that might be different).

Conclusion¶

I had hoped that my science fiction films would be vindicated as being good movies as long as you were one of the people who 'got it', and in that sense this project backfired. Instead I've discovered that the same argument I was planning on using to defend my beloved genre actually works better as a defense of Twilight.

Update: I've written an addendum to this, looking at the data after normalizing it (taking into account the bias of the critics/audiences)¶

comments

social