Measuring Film Quality: Revisiting “Guilty Pleasures”

In February 2019, I posed a deceptively simple question:

What makes a pleasure “guilty?”

To answer this question, I focused on films, specifically those I had seen multiple times. I gathered publicly-available data on these movies in order to assess how these films were regarded by both critics and fans. At that time, there were 557 such films, though I excluded 23 Charlie Chan films, having discussed them in a previous post. With these data, I generated a “perceived quality” (“PQ”) score. By comparing how much I liked a film to its PQ, I compiled a list of 11 “guilty pleasure” films, those I love despite their relatively poor reputations.

Recently, I updated that list, both adding films I forgot to include the first time and watching others for a second time. I also updated all data from the Internet Movie Database (“IMDb”) and the online movie rating site Rotten Tomatoes (“RT”), learning the latter no longer provides a count of the number of RT users – as opposed to “critics” – choosing to rate a film. Instead, they provide a characterization such as “10,000+ ratings;” I removed it from the database.

The goal was less to reexamine guilty pleasures – that list barely changed – than it was to examine the process itself.

**********

On June 5, 2021 – OK, early on the morning of June 6 – I re-watched the 1996 Ted Demme film Beautiful Girls. In so doing, I increased the number of films in the database to 638.

Table 1 summarizes 10 of the remaining 13 variables, excluding title, category (described below) and date first shown in the United States. “Maltin” is the number of stars (BOMB=0) assigned by film critic Leonard Maltin in either his 2003 or 2008 Leonard Maltin’s Movie and Video Guide.[1]

Table 1: Summary statistics for Film Ratings Measures

MeasureNMean (SD*)MedianMinimumMaximum
Year of Release6381973.3 (22.6)198119202019
Length (mins.)638102.5 (19.5)101.048  220
IMDb Score6387.1 (0.7)7.24.29.0
IMDb Raters638104,186.7 (239,116)22,887.51692,359,960
Tomatometer57976.3 (21.2)830100
RT Critic Average5797.0 (1.3)7.02.19.8
RT Critics63848.8 (60.4)340541
RT Audience Score63371.7 (17.8)762096
RT Audience Average6333.8 (0.4)3.82.34.6
Maltin Stars6152.8 (0.7)304

*SD=standard deviation, a measure of how tightly packed values are around the mean: the smaller the value, the tighter the packing. In a normal distribution, 68% of values are within 1 SD, 95% are within 2 SD and 99% are within 3 SD.

Compared to the original 557 films, this set of 638 films:

  • Were released one year earlier (mean and median) – though maximum increased to 2019 (Avengers: Endgame) and SD by 1.7.
  • Were 0.9 minutes shorter, with minimum now 48 minutes (Sherlock Jr.), increasing SD 1.7.
  • Had basically the same average/median IMDb score – a weighted average of user 0-10 ratings – and range, 4.2 (The Adventures of Rocky & Bullwinkle) to 9.0 (12 Angry Men, The Dark Knight).
  • Saw a mean increase of ~25,000 IMDb raters, with median increase of ~3,800; SD increased ~55,000, echoing a wider range of 169 (Southside 1-1000) to 2,359,960 (The Dark Knight), reflecting passage of time and addition of recent films with >1 million IMDb raters: Shutter Island, The Prestige, The Avengers and Inception.
  • Saw average, median and SD of Tomatometer – percentage of RT critics deeming a film “fresh” – drop slightly, with range still 0 (Once Upon a Crime…) to 100 (n=44).
  • Saw no change in average, median and SD of RT critic rating – 0-10 scale – while range widened: 2.1 (Hexed) to 9.8 (Sherlock Jr.).
  • Saw average, median and SD of number of RT critics increase by 8.8, 4.0 and 17.9, respectively, reflecting widened range of 0 (Charlie Chan at the Race Track, Murder in the Big House) to 541 (Avengers: Endgame)
  • Brought no change to average, median and SD for RT Audience Score – Tomatometer for fans – while range increased slightly: 17 (Street of Chance) to 97 (12 Angry Men).
  • Saw average and median Audience Rating – 0-5 scale – increase by 0.3: the distribution simply shifted to the right, with same SD and new range of 2.3 (Rocky & Bullwinkle, The Opposite Sex and How to Live With Them) to 4.6 (n=5).
  • Brought no change to Maltin statistics – though BOMB films increased from four to five with addition of Yellowbeard.
  • Basically, while the films I added were slightly older and shorter on average, and raters increased with time, the distribution of ratings did not materially change with the addition of 81 movies and updated data – except RT Audience Ratings, which increased for most movies.

Put another way, the “median” film I have seen multiple times remains a good-but-not-spectacular film like the 1980 comedy 9 to 5, which is 109 minutes long, has an IMDb score of 6.9 from just over 29,000 rates, a Tomatometer of 83 with an average 7.0 rating from 41 critics, a RT Audience Score of 83 with an average 3.8 rating, and 3.5 stars from Maltin. Half of the films I have seen multiple times are better-rated overall than 9 to 5 and half are worse-rated.

But here we run into the problem I sidestepped in the original post – how to create a single quality measure for all 638 films, when I only have complete data for 561 (87.9%) of them, a decrease of 4.6 percentage points. The biggest problem is lackof Tomatometer and RT Critic Rating data for 59 films with fewer than five critics.

**********

Missing data is perhaps the bête noire of data analysts. There are only two solutions: 1) only use cases with complete data or 2) use statistical methods to estimate missing cases. The first solution is reasonable if you are not trying to generate a single value for every case and/or data are missing at random. Clearly this solution will not work here. Not only do we want a single value for each film – and only using the five variables (year, runtime, IMDb score, number IMDb raters, number RT critics) with no missing data feels inadequate – there is a clear pattern to the missing data. For example, 17 of the 23 films lacking Matlin stars were released after 2008. Conversely, the five films lacking RT audience data were released between 1936 and 1952, while the 59 films missing RT critic data have release year and runtime of 1947 and 80.6 minutes, respectively, with all but four released between 1931 and 1957.

This leaves estimation. A straightforward way is to use ordinary least squares regression (“OLS”), analogous to Y = Slope*X + “Where line crosses Y-axis”, or y = m*x + b, the formula we learned when first plotting data points. OLS regression similarly estimates how a dependent variable – say, RT Audience Score – is related to one or more independent variables – say, the five film measures with complete data.

Before we begin, however, let us set a baseline to see how well the estimation process worked: the correlation matrix derived from the 561 films with complete data. I characterize these correlations in the previous post, so I do not belabor them here.

Figure 1: Film Quality Measures Correlations Matrix (n=561)

Missing data estimation is an iterative process. I progressed from variables with the fewest missing cases (5 each for RT Audience Score and Average) to those with the most missing cases (59 for Tomatometer, RT Critic Average). Consecutive OLS regressions are summarized in Table 2; Intercooled Stata 9.2 (“Stata”)[2] was used for all analyses. Because Maltin Stars is an ordinal variable – it has seven discrete categories, rather than being continuous – I considered using ordinal logistic regression, but the OLS model proved a better fit; estimated Maltin Stars were rounded to the nearest “half-star.”

Table 2: Iterative Film Quality Measure OLS Regressions

VariableRT Aud ScoreRT Aud AveMaltin StarsRT Critic AveTomato- meter
Constant-532.6751.95211.71015.912-79.111
Year of Release0.217-0.00005-0.007-0.0080.043
Length (mins.)0.0020.00060.004-0.002-0.033
IMDb Score25.0980.0370.4880.6771.576
IMDb Raters-0.000027.7e-8-8.8e-75.0e-8-4.4e-6
RT Critic Average    14.858
RT Critics0.001-0.00020.0020.002-0.017
RT Audience Score 0.022-0.0040.0250.434
RT Audience Average  0.298-0.268-19.504
Maltin Stars   0.5251.126
Number of cases633633615579579
Adjusted R-squared0.8090.9170.4610.7730.855

These are solid models, accounting for between 46.1 (Maltin Stars) and 91.7% (RT Audience Average) of a dependent variable’s variance. Moreover, the estimated values had high face validity – they made “sense.” And as Figure 2 shows, the 151 estimated cases barely changed relationships between these variables.

Figure 2: Film Quality Measures Correlations Matrix (n=638)

On average, the 45 correlations increased 0.045, with IMDb Score’s 9 correlations increasing a mean 0.013, and Runtime and Year of Release increasing by 0.086 and 0.134, respectively. The latter increases reflect the preponderance of missing data among both older/shorter and newer/longer films – SD increased, increasing all 18 correlations. The story is the same if you look at the absolute value of changes: an average shift of 0.050 in either direction, with Tomatometer’s 9 correlations changing by a mean of 0.020.

In other words, this data estimation process was very successful.

**********

Armed with complete data, I used factor analysis[3] to calculate a single “film quality” score. The results were nearly identical to those from the previous post: using all 10 variables yields two factors accounting for 68.1% (PQ) and 26.0% (Public Awareness) of the total variance, while removing year of release, runtime and numbers of raters yields one factor (PQ) accounting for an astonishing 92.6% of the total variance: IMDb users, RT critics, RT fans and Maltin rate movies in remarkably similar ways. Table 3 presents the factor loadings (correlations with underlying dimension being assessed) and score coefficients used to generate a single PQ score.

Table 3: Factor Analysis of Film Quality Measures, Two Iterations

VariableAll 10 Variables 68.2%“Rating” Variables Only 92.6%
 LoadingsCoefficientsLoadingsCoefficients
Year of Release-0.1720.013  
Length (mins.)0.2480.017  
IMDb Score0.9350.2270.9140.104
IMDb Raters0.4250.035  
Tomatometer0.8440.0990.8690.142
RT Critic Average0.9210.3110.9320.339
RT Critics0.3910.059  
RT Audience Score0.9190.1940.9230.253
RT Audience Average0.9030.1710.9030.203
Maltin Stars0.6790.0380.6910.037

From these coefficients, Stata[4] calculated two PQ scores – PQAll and PQRating – for each film. Think of these values as SD above or below 9 to 5.  They are correlated a whopping 0.992, though they do have subtle differences, as Table 4 reveals:

Table 4: 30 Highest Rated Films I Have Seen Multiple Times, Compared by PQ Score

PQAllPQScore
30. The Apartment30. The Maltese Falcon (1941)
29. Memento29. The Cabinet of Dr. Caligari
28. No Country For Old Men28. Witness for the Prosecution
27. Some Like It Hot27. L.A. Confidential
26. Sherlock Jr.26. The Apartment
25. Vertigo25. Vertigo
24. The Wizard of Oz24. Annie Hall
23. The Third Man23. Back to the Future
22. L.A. Confidential22. The General
21. The Avengers21. Kind Hearts and Coronets
20. Double Indemnity20. The Wizard of Oz
19. On the Waterfront19. Metropolis
18. M (1931)18. Some Like It Hot
17. Metropolis17. North By Northwest
16. North By Northwest16. Star Wars Episode IV: A New Hope
15. Chinatown15. Double Indemnity
14. Back to the Future14. Chinatown
13. Sunset Boulevard13. The Third Man
12. Rear Window12. On the Waterfront
11. Citizen Kane11. Rear Window
10. It’s A Wonderful Life10. It’s a Wonderful Life
9. Psycho9. M (1931)
8. Indiana Jones and the Raiders of the Lost Ark8. Psycho
7. Star Wars Episode IV: A New Hope7. Citizen Kane
6. Casablanca6. Indiana Jones and the Raiders of the Lost Ark
5. Inception5. Sunset Boulevard
4. 12 Angry Men4. Sherlock Jr.
3. Avengers: Endgame3. Pulp Fiction
2. Pulp Fiction2. Casablanca
1. The Dark Knight1. 12 Angry Men

Just 24 films appear on both lists (including L.A. Confidential, my favorite movie). Six newer (median 2009), longer (146.5 minutes), more-oft-rated films (1,214,741 IMDb raters, 352 RT critics) films in the PQAll Top 30 are replaced by six older (1945), shorter (96.5 minutes), slightly-less-oft-rated films (131,469; 58.5). Only six films – It’s A Wonderful Life, Psycho, Indiana Jones and the Raiders of the Lost Ark, Casablanca, 12 Angry Men and Pulp Fiction – rank in the top 10 on both PQ scores.

Still, if I were to choose a set of recent films likeliest still to be highly regarded a few decades from now, Memento, No Country for Old Men, The Avengers, Inception, Avengers: Endgame and The Dark Knight are an excellent starting point. Three of them, along with The Prestige and Batman Begins, were directed by Christopher Nolan – perhaps the best director of our time.

While this is a very impressive list of films – I was even more blown away by 12 Angry Men the second time – it is based ONLY on films I have seen multiple times. It excludes highly-regarded films (per the 20 top-ranked films on IMDb) I have only seen once: The Shawshank Redemption, first two Godfather films, Fight Club, Forrest Gump, Star Wars Episode V: The Empire Strikes Back, The Matrix, Goodfellas, One Flew Over the Cuckoo’s Nest and Se7en. And then there are movies I have not seen at all: Schindler’s List; the Lord of the Rings trilogy; The Good, the Bad and the Ugly; and Seven Samurai.

In fact, I have only seen four of the top 20 IMDb-rated movies multiple times: The Dark Knight, 12 Angry Men, Pulp Fiction and Inception. This reflects my personal taste in movies: older, noir-tinged, mysteries and comedies rather than more contemporary fantasy, war-based or western films; whereas the median year of release of the top 250 films by PQScore is 1971.5, respectively, that of the 250 films by IMDb score is 1994 – with 51 being released in 2011 or later, a strong indication of recency bias in the IMDb score data.

Another way to consider my particular taste in movies is to examine the distribution of year of release (Figure 3). There are two distinct peaks –1946-50 (n=73, 11.4%) and 1982-98 (n=256, 40.1%). The former period roughly corresponds to the pinnacle of classic film noir, while the latter is my primary movie-attending period – ages 15 to 32. Indeed, 119 (18.7%) of these films are classic-era films noir, released between 1940 (Stranger on the Third Floor) and 1958 (Touch of Evil) with average PQAll and PQScore of 0.03 and 0.08, respectively (relative to 0, overall). This excludes 18 films directed by Alfred Hitchcock, 10 of which are widely considered film noir, with average PQAll and PQScore of 0.92and 0.94, respectively. Finally, there are 49 films released before 1960 – not film noir or Hitchcock, not Charlie Chan (n=23; both -0.66), starring The Marx Brothers (n=9; 0.38, 0.48) – with averages of 0.92 and 0.97, respectively. The other 395 films (61.9%) – excluding the 25 directed by Woody Allen[5] (n=25; 0.26, 0.30) – have averages of -0.15 and -0.18, respectively. These values are broadly similar to those from the previous post, excepting the addition of “Charlie Chan.”

Figure 3: Distribution of Year of Release is Bimodal

Even more instructive is to compare my favorite film by time period (>5 films) – or, at least, my best guess absent a formal assessment – to the film with the highest PQAll and PQScore from that period.

1920 to 1930

Number films = 9

Average PQAll = 1.12

Average PQScore = 1.20

Top film: Metropolis (1.61, 1.53)

Personal favorite: The Phantom of the Opera (0.76, 0.84), albeit barely

Comment: Phantom has the lowest PQ scores of the nine films released in the 1920s I have seen multiple times, revealing their overall quality.

1931

Number films = 5

Average PQAll = 0.65

Average PQScore = 0.74

Top film: M (1.61,1.65)

Personal favorite: M

Comment: I have seen at least five other films released in 1931, making it a key year in my personal fandom

1932 to 1934

Number films = 5

Average PQAll = 0.72

Average PQScore = 0.83

Top film: The Thin Man (1.32,1 .43)

Personal favorite: The Thin Man

1935

Number films = 5

Average PQAll = 0.27

Average PQScore = 0.34

Top film: A Night at the Opera (1.23, 1.30)

Personal favorite: A Night at the Opera

1936 to 1939

Number films = 29

Average PQAll = -0.18

Average PQScore = -0.17

Top film: The Wizard of Oz (1.57, 1.50)

Personal favorites: After the Thin Man (0.83, 0.94), Charlie Chan at Treasure Island (-0.18, -0.15), Charlie Chan in Reno (-0.62, -0.64)

Comment: For me, the 1930s combine genuinely great films – eight from the Marx Brothers – with 16 Charlie Chan films.

1940

Number films = 10

Average PQAll = 0.19

Average PQScore = 0.24

Top film: Rebecca (1.43, 1.41)

Personal favorite: Foreign Correspondent (0.66, 0.72), though Rebecca and His Girl Friday (1.28, 1.35) are wicked close

Comment: Hitchcock made his first Hollywood films in 1940, and they were knockouts…though the best was yet to come.

1941

Number films = 11

Average PQAll = 0.19

Average PQScore = 0.23

Top film: Citizen Kane (1.75, 1.66)

Personal favorite: The Maltese Falcon (1.38, 1.44)

Comment: Film noir bursts on the scene with a bang.

1942

Number films = 9

Average PQAll = 0.04

Average PQScore = 0.07

Top film: Casablanca (1.88, 1.77)

Personal favorite: All Through the Night (0.15, 0.26)

Comment: …as does Humphrey Bogart.

1943 to 1944

Number films = 10

Average PQAll = 0.64

Average PQScore = 0.70

Top film: Double Indemnity (1.61, 1.59)

Personal favorite: Laura (1.25, 1.31)

Comment: 1944 is the best year for film quality (0.74, 0.79) since 1931

1945

Number films = 7

Average PQAll = -0.25

Average PQScore = -0.20

Top film: Scarlet Street (0.82, 0.91)

Personal favorite: Spellbound (0.40, 0.43)

Comment: Followed by a mediocre 1945.

1946

Number films = 17

Average PQAll = 0.01

Average PQScore = 0.05

Top film = It’s A Wonderful Life (1.77, 1.65)

Personal favorite: Deadline at Dawn (-0.56, -0.50)

Comment: This is the first large disconnect between the “best” film released in a year and my favorite film from that year.

1947

Number films = 19

Average PQAll = -0.18

Average PQScore = -0.22

Top film: Out of the Past (1.77, 1.65)

Personal favorite: Out of the Past

1948

Number films = 13

Average PQAll = 0.30

Average PQScore = 0.38

Top film: Rope (0.91, 0.94)

Personal favorites: Call Northside 777 (0.09, 0.09) and The Naked City (0.39, 0.43)

1949

Number films = 11

Average PQAll = 0.26

Average PQScore = 0.30

Top film = The Third Man (1.58, 1.62)

Personal favorite: Impact (-0.60, -0.65)

1950

Number films = 13

Average PQAll = 0.08

Average PQScore = 0.12

Top film: Sunset Boulevard (1.70, 1.69)

Personal favorite: Where the Sidewalk Ends (0.59, 0.71)

Comment: Classic film noir and Hitchcock yielded some of the best films ever made.

1951

Number films = 8

Average PQAll = 0.20

Average PQScore = 0.28

Top film: Strangers on a Train (1.27, 1.31)

Personal favorite: The Enforcer (-0.19, -0.15)

1952

Number films = 6

Average PQAll = -0.05

Average PQScore = -0.01

Top film: The Narrow Margin (0.72, 0.85)

Personal favorite: Kansas City Confidential (-0.05, -0.02)

Comment: This is the first time the best film I have seen multiple times released that year is far from the best film released that year. That honor goes to Singin’ in the Rain, a movie I have yet to see in its entirety.

1953

Number films = 7

Average PQAll = -0.02

Average PQScore = 0.02

Top film: The Big Heat (1.04, 1.12)

Personal favorite: 99 River Street (0.35, 0.47)

1954

Number films = 7

Average PQAll = 0.40

Average PQScore = 0.38

Top film: Rear Window (1.72, 1.64)

Personal favorite: Dial M For Murder (0.88, 0.84), edging out Rear Window

Comment: I have yet to see Seven Samurai…and Hitchcock has hit his absolute peak.

1955

Number films = 7

Average PQAll = 0.77

Average PQScore = 0.85

Top films: Diabolique (1.25, 1.27) and Rififi (1.24, 1.27)

Personal favorite: Muerte de un ciclista (Death of a Cyclist) (0.70, 0.82)

Comment: For the first time since 1920-31, foreign films dominate – with Germany replaced by France and Spain. It also marks the shift the overseas shift of film noir, presaging La Nouvelle Vague.

1956 to 1959

Number films = 15

Average PQAll = 0.94

Average PQScore = 0.94

Top film: 12 Angry Men (1.95, 1.80)

Personal favorite: 12 Angry Men

1960-69

Number films = 26

Average PQAll = 0.31

Average PQScore = 0.33

Top film: Psycho (1.79, 1.66)

Personal favorite: The Apartment (1.49, 1.44), though Psycho is close.

Comment: I have not seen multiple times any films released in 1961, 1969 or 1970. And most of the 1960s is a cinematic wasteland for me…though I may have seen Dr. Strangelove, or How I Stopped Worrying and Learned to Love the Bomb more than once.

1970 to 1972

Number films = 6

Average PQAll = 0.36

Average PQScore = 0.39

Top film = Sleuth (1.17, 1.19)

Personal favorites: What’s Up, Doc? (0.70, 0.76) and Willie Wonka and the Chocolate Factory (0.79, 0.79)

Comment: The Godfather – which I have only seen once – is the best film released in 1972. Some would argue…ever.

1973

Number films = 8

Average PQAll = 0.44

Average PQScore = 0.46

Top films:  Paper Moon (1.32, 1.36) and The Sting (1.37, 1.31)

Personal favorite: Charley Varrick (0.45, 0,49) edges The Sting and American Graffiti (0.83, 0.90).

1974 to 1975

Number films = 11

Average PQAll = 0.47

Average PQScore = 0.51

Top film:  Chinatown (1.66, 1.61)

Personal favorite: Murder on the Orient Express (0.41, 0.43)

Comment: Chinatown is a masterpiece; The Godfather: Part 2 (only seen once) is considered better.

1976 to 1977

Number films = 10

Average PQAll = 0.19

Average PQScore = 0.20

Top film: Star Wars Episode IV: A New Hope (1.87, 1.57)

Personal favorites: Murder by Death (-0.06, -0.05) and The Seven-Per-Cent Solution (-0.46, -0.39)

1978

Number films = 11

Average PQAll = -0.64

Average PQScore = -0.61

Top film: Superman (0.81, 0.85)

Personal favorites: Death on the Nile (0.04, 0.03) and Thank God It’s Friday (-2.26, -2.24)

Comment: I genuinely do not understand why the charming Friday does get more love. And as solid as Superman is, The Deer Hunter (which I have not seen) is probably the best film released in 1978.

1979

Number films = 12

Average PQAll = 0.06

Average PQScore = 0.11

Top films: Being There (1.20, 1.20) and Manhattan (1.19, 1.21)

Personal favorites: Manhattan and The In-Laws (0.50, 0.61)

1980

Number films = 12

Average PQAll = -0.22

Average PQScore = -0.18

Top film: Airplane! (1.02, 1.06)

Personal favorite: Times Square (-0.60, -0.55)

Comment: I have only seen Empire Strikes Back once.

1981

Number films = 11

Average PQAll = 0.23

Average PQScore = 0.29

Top film: Indiana Jones and the Raiders of the Lost Ark

Personal favorite: Indiana Jones and the Raiders of the Lost Ark

1982

Number films = 17

Average PQAll = 0.06

Average PQScore = 0.09

Top film: Blade Runner (1.30, 1.14)

Personal favorites: Fast Times at Ridgemont High (0.13, 0.15) and Hammett (-0.60, -0.52)

Comment: Wow, Harrison Ford dominated movies from 1977 and 1983.

1983

Number films = 9

Average PQAll = -0.18

Average PQScore = -0.12

Top film: A Christmas Story (1.14, 1.15)

Personal favorite: Valley Girl (-0.22, -0.06)

1984

Number films = 19

Average PQAll = -0.34

Average PQScore = -0.30

Top film: This is Spinal Tap (1.22, 1.26)

Personal favorite: The Cotton Club (-0.67, -0.66)

Comment: We are now squarely in the age of mediocre films I saw in the theater as a teenager/young adult then chose to watch again. And maybe I need to see Amadeus again.

1985

Number films = 14

Average PQAll = 0.22

Average PQScore = 0.24

Top film: Back to the Future (1.70, 1.48)

Personal favorite: Back to the Future, though The Sure Thing is close (0.09, 0.18)

1986

Number films = 17

Average PQAll = -0.16

Average PQScore = -0.14

Top films: Hannah and Her Sisters (1.11, 1.12) and Blue Velvet (1.07, 1.08)

Personal favorite: While Legal Eagles (-1.73, -1.83) is a top guilty pleasure, I may like Hannah more.

Comment: Yes, I have never seen Aliens.

1987

Number films = 17

Average PQAll = -0.33

Average PQScore = -0.32

Top film: Wings of Desire (1.39, 1.41)

Personal favorite: House of Games (0.57, 0.67)

1988

Number films = 13

Average PQAll = -0.43

Average PQScore = -0.41

Top film: Die Hard (1.48, 1.33)

Personal favorite: Who Framed Roger Rabbit (0.97, 0.97)

1989

Number films = 13

Average PQAll = -0.49

Average PQScore = -0.49

Top film: Crimes and Misdemeanors (0.98, 1.00)

Personal favorite: Forced to choose from a lot of meh, I pick The Big Picture (-0.84, -0.77) for its charming cast.

1990

Number films = 13

Average PQAll = -0.32

Average PQScore = -0.29

Top film: Metropolitan (0.58, 0.66)

Personal favorite: Metropolitan

Comment: The one film released in 1990 in the IMDb Top 250 is Goodfellas, which I look forward to rewatching. [Ed. note: I neglected to add Awakenings, which I have seen twice, and which tops Metropolitan among films I have seen multiple times – though the latter is still my personal favorite among these group.]

1991

Number films = 20

Average PQAll = -0.70

Average PQScore = -0.73

Top film: JFK (0.99, 0.85)

Personal favorite: If I have to pick one from this meh collection – Dead Again (0.15, 0.23)

Comment: I should watch The Silence of the Lambs again.

1992

Number films = 16

Average PQAll = -0.63

Average PQScore = -0.64

Top film: The Player (0.94, 0.98)

Personal favorite: The Public Eye (-0.99, -1.01)

Comment: I should watch Reservoir Dogs again – as we hit rock bottom in the early 1990s.

1993

Number films = 13

Average PQAll = -0.85

Average PQScore = -0.85

Top film: The Fugitive (1.01, 0.95)

Personal favorite: Manhattan Murder Mystery (0.34, 0.40)

Comment: I spoke too soon…yeesh. Perhaps once I finally see Schindler’s List. [Ed. note: A few days after posting this, I watched Dazed and Confused for the second time, and I think it replaces both The Fugitive and Manhattan Murder Mystery – making it both the “best” film and my favorite film release in 1993 I have seen multiple times. I have also seen Jurassic Park only once.]

1994

Number films = 18

Average PQAll = -0.32

Average PQScore = -0.35

Top film: Pulp Fiction (2.16, 1.72)

Personal favorite: The Shadow (-1.63, -1.74) – easily the widest gap between “best” and “favorite”

Comment: Do not be fooled by these data. Hollywood has now entered an absolute golden age, rivaling its best years. Remember, I have only seen Shawshank Redemption, Forrest Gump and the phenomenal Léon: The Professional once. And I have never seen The Lion King. These four movies and Pulp Fiction are in the top 40 films by IMDb score.

1995

Number films = 12

Average PQAll = -0.28

Average PQScore = -0.34

Top film: The Usual Suspects (2.16, 1.72)

Personal favorite: The Usual Suspects

Comment: Eight of the IMDb Top 250 were released in 1995.

1996

Number films = 15

Average PQAll = -0.36

Average PQScore = -0.36

Top film: Fargo (1.42, 1.31)

Personal favorite: Big Night (0.69, 0.78), probably.

1997

Number films = 9

Average PQAll = 0.24

Average PQScore = 0.20

Top film: L.A. Confidential (1.59, 1.44)

Personal favorite: L.A. Confidential

Comment: I am still upset L.A. Confidential did not win the Academy Award for Best Picture…and Titanic is not even in the IMDb Top 250; L.A. Confidential only ranks 5th.

1998

Number films = 16

Average PQAll = -0.24

Average PQScore = -0.26

Top film: Rushmore (1.08, 1.07)

Personal favorites: Dark City (0.45, 0.37) and Pleasantville (0.54, 0.46)

Comment: I have zero interest in Saving Private Ryan – in fact, Steven Spielberg does almost nothing for me. There, I said it. And I just realized I left The Big Lebowski off my list – though it would not rank much higher than Rushmore.

1999

Number films = 14

Average PQAll = -0.40

Average PQScore = -0.51

Top film: Toy Story 2 (1.34, 1.19)

Personal favorites: Cradle Will Rock (-0.24, -0.29) and Mystery Men (-0.91, -0.96)

Comment: I may watch Fight Club and The Matrix again at some point.

2000 to 2001

Number films = 12

Average PQAll = 0.35

Average PQScore = 0.21

Top film: Memento (1.52, 1.19)

Personal favorite: Mulholland Drive (1.03, 0.79)

2002-09

Number films = 20

Average PQAll = 0.19

Average PQScore = -0.14

Top film: The Dark Knight (2.19, 1.39)

Personal favorite: The Dark Knight

Comment: For the first time, the difference in scores reflects the bias in PQAll toward longer, more recent films with many raters.

2010-19

Number films = 12

Average PQAll = 1.09

Average PQScore = 0.72

Top film: Avengers: Endgame (2.03, 1.30) and Inception (1.95, 1.17)

Personal favorite: Predestination (0.28, 0.10)

Comment: We come full circle: as in the 1920s, the handful of films released in the 2010s I have seen multiple times are generally well-regarded. And my favorite (sorry, Endgame, Hugo, Doctor Strange) is again least well-regarded, but still better than average.

**********

Figure 4 reinforces the year-by-year analysis. It shows early and late peaks: one for the 1920s because I carefully chose the best films to watch and rewatch, and one for the 2010s, because I stopped seeing movies in theatres first, choosing the best films to watch (OK, and all 23 Marvel Cinematic Universe films) then watch again. Moreover, average PQScore is higher than average PQAll until about 1940 – when it basically draws even until the early 1990s – because the latter elevates longer, recent, heavily-reviewed films. Around 2000, average PQAll pulls much further ahead for the same reason.

After a sharp decline in both values through the 1930s (excepting a slight uptick in 1931) – reflecting the Fox Charlie Chan films I love more than most – the emergence of film noir around 1940 pulls values up again. They generally stay above 0 through the early 1950s, peaking sharply in 1944. A preponderance of exceptional Hitchcock and foreign films sends scores skyrocketing in the late 1950s, with a lower peak from the 1960s into the mid-1970s: once again a lack of films elevates scores.

Figure 4: Average PQAll and PQScores by Year of Release/Midpoint of Range of Year of Release, 1920-2019

Almost by definition, I first saw any film released through the mid-1970s, not in a movie theater, but on television, through one of Yale’s films societies or through a rental/streaming service. This will inevitably bias toward “better” films. That changed in the late 1970s, when I began regularly seeing films for the first time in a movie theater – resulting in a much wider range of quality. The steep decline in scores in 1978 shows that, as does the zig-zagging around 0 through the mid-1980s. I cannot really explain why scores plummet in the late 1980s and early 1990s, though there is evidence overall film quality was much lower in those year.

The sharp spike up in 1994 masks an even more dramatic increase in overall film quality. I have long thought the peace and prosperity of the last six years of the Clinton Administration yielded a new golden age in American cinema; perhaps studio executives felt freer to experiment with original screenplays. I also though that changed after the 9-11, but…maybe not. Still, I suspect the preponderance of post-2000 films in the IMDb Top 250 reflects recency bias more than actual quality.

Basically, while I consider myself a cinephile, like everyone who watches movies my tastes range from genuine works of art to the guiltiest of guilty pleasures. And that is how it is supposed to be – we like what we like, not what we are “supposed” to like. Still, whether I am simply more honest in admitting how much I genuinely like the Fox Charlie Chan films, and films like Deadline at Dawn, Impact, The Seven-Per-Cent Solution, Thank God It’s Friday, Times Square, Hammett, The Cotton Club, Legal Eagles, The Big Picture, The Public Eye, The Shadow and Mystery Men remains to be seen.

In the meantime, I just shared Cat People (1942) with my wife Nell – who quite enjoyed it – so I need to update my Excel workbook. Again.

Until next time…


[1] New York, NY: New American Library

[2] StataCorp. 2005. Stata Statistical Software: Release 9. College Station, TX: StataCorp LP.

[3] Specifically: factor analysis, principal factors, varimax rotation, forcing one or two factors, depending on input variables.

[4] Using “Predict” command in Stata

[5] Despite my ambivalence about Allen as a human being, I still love many of his films.

Published by

Matt Berger

I am a data geek, writer, investigator and film noir devotee with academic training in political science (Yale BA, Harvard MA), biostatistics (Boston University SPH MA) and epidemiology (Boston University SPH PhD). In January 2021, I finished writing Interrogating Memory: Film Noir Spurs a Deep Dive Into My Family History...and My Own for which I seek literary representation and a publisher. In Chapter 6: So...What Is Film Noir, Again?, I analyze my film noir database, which contains 4,825 titles. My musical holy trinity is Genesis, Miles Davis and Stan Ridgway. I am a liberal Democrat who understands a thriving democracy requires at least two mature political parties. I grew up just outside Philadelphia, and I live just outside Boston, where my wife and daughters keep me happy, sane and grounded. Please ask me anything else you want to know.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s