In February 2019, I posed a deceptively simple question:
What makes a pleasure “guilty?”
To answer this question, I focused on films, specifically those I had seen multiple times. I gathered publicly-available data on these movies in order to assess how these films were regarded by both critics and fans. At that time, there were 557 such films, though I excluded 23 Charlie Chan films, having discussed them in a previous post. With these data, I generated a “perceived quality” (“PQ”) score. By comparing how much I liked a film to its PQ, I compiled a list of 11 “guilty pleasure” films, those I love despite their relatively poor reputations.
Recently, I updated that list, both adding films I forgot to include the first time and watching others for a second time. I also updated all data from the Internet Movie Database (“IMDb”) and the online movie rating site Rotten Tomatoes (“RT”), learning the latter no longer provides a count of the number of RT users – as opposed to “critics” – choosing to rate a film. Instead, they provide a characterization such as “10,000+ ratings;” I removed it from the database.
The goal was less to reexamine guilty pleasures – that list barely changed – than it was to examine the process itself.
**********
On June 5, 2021 – OK, early on the morning of June 6 – I re-watched the 1996 Ted Demme film Beautiful Girls. In so doing, I increased the number of films in the database to 638.
Table 1 summarizes 10 of the remaining 13 variables, excluding title, category (described below) and date first shown in the United States. “Maltin” is the number of stars (BOMB=0) assigned by film critic Leonard Maltin in either his 2003 or 2008 Leonard Maltin’s Movie and Video Guide.[1]
Table 1: Summary statistics for Film Ratings Measures
| Measure | N | Mean (SD*) | Median | Minimum | Maximum |
| Year of Release | 638 | 1973.3 (22.6) | 1981 | 1920 | 2019 |
| Length (mins.) | 638 | 102.5 (19.5) | 101.0 | 48 | 220 |
| IMDb Score | 638 | 7.1 (0.7) | 7.2 | 4.2 | 9.0 |
| IMDb Raters | 638 | 104,186.7 (239,116) | 22,887.5 | 169 | 2,359,960 |
| Tomatometer | 579 | 76.3 (21.2) | 83 | 0 | 100 |
| RT Critic Average | 579 | 7.0 (1.3) | 7.0 | 2.1 | 9.8 |
| RT Critics | 638 | 48.8 (60.4) | 34 | 0 | 541 |
| RT Audience Score | 633 | 71.7 (17.8) | 76 | 20 | 96 |
| RT Audience Average | 633 | 3.8 (0.4) | 3.8 | 2.3 | 4.6 |
| Maltin Stars | 615 | 2.8 (0.7) | 3 | 0 | 4 |
*SD=standard deviation, a measure of how tightly packed values are around the mean: the smaller the value, the tighter the packing. In a normal distribution, 68% of values are within 1 SD, 95% are within 2 SD and 99% are within 3 SD.
Compared to the original 557 films, this set of 638 films:
- Were released one year earlier (mean and median) – though maximum increased to 2019 (Avengers: Endgame) and SD by 1.7.
- Were 0.9 minutes shorter, with minimum now 48 minutes (Sherlock Jr.), increasing SD 1.7.
- Had basically the same average/median IMDb score – a weighted average of user 0-10 ratings – and range, 4.2 (The Adventures of Rocky & Bullwinkle) to 9.0 (12 Angry Men, The Dark Knight).
- Saw a mean increase of ~25,000 IMDb raters, with median increase of ~3,800; SD increased ~55,000, echoing a wider range of 169 (Southside 1-1000) to 2,359,960 (The Dark Knight), reflecting passage of time and addition of recent films with >1 million IMDb raters: Shutter Island, The Prestige, The Avengers and Inception.
- Saw average, median and SD of Tomatometer – percentage of RT critics deeming a film “fresh” – drop slightly, with range still 0 (Once Upon a Crime…) to 100 (n=44).
- Saw no change in average, median and SD of RT critic rating – 0-10 scale – while range widened: 2.1 (Hexed) to 9.8 (Sherlock Jr.).
- Saw average, median and SD of number of RT critics increase by 8.8, 4.0 and 17.9, respectively, reflecting widened range of 0 (Charlie Chan at the Race Track, Murder in the Big House) to 541 (Avengers: Endgame)
- Brought no change to average, median and SD for RT Audience Score – Tomatometer for fans – while range increased slightly: 17 (Street of Chance) to 97 (12 Angry Men).
- Saw average and median Audience Rating – 0-5 scale – increase by 0.3: the distribution simply shifted to the right, with same SD and new range of 2.3 (Rocky & Bullwinkle, The Opposite Sex and How to Live With Them) to 4.6 (n=5).
- Brought no change to Maltin statistics – though BOMB films increased from four to five with addition of Yellowbeard.
- Basically, while the films I added were slightly older and shorter on average, and raters increased with time, the distribution of ratings did not materially change with the addition of 81 movies and updated data – except RT Audience Ratings, which increased for most movies.
Put another way, the “median” film I have seen multiple times remains a good-but-not-spectacular film like the 1980 comedy 9 to 5, which is 109 minutes long, has an IMDb score of 6.9 from just over 29,000 rates, a Tomatometer of 83 with an average 7.0 rating from 41 critics, a RT Audience Score of 83 with an average 3.8 rating, and 3.5 stars from Maltin. Half of the films I have seen multiple times are better-rated overall than 9 to 5 and half are worse-rated.
But here we run into the problem I sidestepped in the original post – how to create a single quality measure for all 638 films, when I only have complete data for 561 (87.9%) of them, a decrease of 4.6 percentage points. The biggest problem is lackof Tomatometer and RT Critic Rating data for 59 films with fewer than five critics.
**********
Missing data is perhaps the bête noire of data analysts. There are only two solutions: 1) only use cases with complete data or 2) use statistical methods to estimate missing cases. The first solution is reasonable if you are not trying to generate a single value for every case and/or data are missing at random. Clearly this solution will not work here. Not only do we want a single value for each film – and only using the five variables (year, runtime, IMDb score, number IMDb raters, number RT critics) with no missing data feels inadequate – there is a clear pattern to the missing data. For example, 17 of the 23 films lacking Matlin stars were released after 2008. Conversely, the five films lacking RT audience data were released between 1936 and 1952, while the 59 films missing RT critic data have release year and runtime of 1947 and 80.6 minutes, respectively, with all but four released between 1931 and 1957.
This leaves estimation. A straightforward way is to use ordinary least squares regression (“OLS”), analogous to Y = Slope*X + “Where line crosses Y-axis”, or y = m*x + b, the formula we learned when first plotting data points. OLS regression similarly estimates how a dependent variable – say, RT Audience Score – is related to one or more independent variables – say, the five film measures with complete data.
Before we begin, however, let us set a baseline to see how well the estimation process worked: the correlation matrix derived from the 561 films with complete data. I characterize these correlations in the previous post, so I do not belabor them here.
Figure 1: Film Quality Measures Correlations Matrix (n=561)
Missing data estimation is an iterative process. I progressed from variables with the fewest missing cases (5 each for RT Audience Score and Average) to those with the most missing cases (59 for Tomatometer, RT Critic Average). Consecutive OLS regressions are summarized in Table 2; Intercooled Stata 9.2 (“Stata”)[2] was used for all analyses. Because Maltin Stars is an ordinal variable – it has seven discrete categories, rather than being continuous – I considered using ordinal logistic regression, but the OLS model proved a better fit; estimated Maltin Stars were rounded to the nearest “half-star.”
Table 2: Iterative Film Quality Measure OLS Regressions
| Variable | RT Aud Score | RT Aud Ave | Maltin Stars | RT Critic Ave | Tomato- meter |
| Constant | -532.675 | 1.952 | 11.710 | 15.912 | -79.111 |
| Year of Release | 0.217 | -0.00005 | -0.007 | -0.008 | 0.043 |
| Length (mins.) | 0.002 | 0.0006 | 0.004 | -0.002 | -0.033 |
| IMDb Score | 25.098 | 0.037 | 0.488 | 0.677 | 1.576 |
| IMDb Raters | -0.00002 | 7.7e-8 | -8.8e-7 | 5.0e-8 | -4.4e-6 |
| RT Critic Average | 14.858 | ||||
| RT Critics | 0.001 | -0.0002 | 0.002 | 0.002 | -0.017 |
| RT Audience Score | 0.022 | -0.004 | 0.025 | 0.434 | |
| RT Audience Average | 0.298 | -0.268 | -19.504 | ||
| Maltin Stars | 0.525 | 1.126 | |||
| Number of cases | 633 | 633 | 615 | 579 | 579 |
| Adjusted R-squared | 0.809 | 0.917 | 0.461 | 0.773 | 0.855 |
These are solid models, accounting for between 46.1 (Maltin Stars) and 91.7% (RT Audience Average) of a dependent variable’s variance. Moreover, the estimated values had high face validity – they made “sense.” And as Figure 2 shows, the 151 estimated cases barely changed relationships between these variables.
Figure 2: Film Quality Measures Correlations Matrix (n=638)
On average, the 45 correlations increased 0.045, with IMDb Score’s 9 correlations increasing a mean 0.013, and Runtime and Year of Release increasing by 0.086 and 0.134, respectively. The latter increases reflect the preponderance of missing data among both older/shorter and newer/longer films – SD increased, increasing all 18 correlations. The story is the same if you look at the absolute value of changes: an average shift of 0.050 in either direction, with Tomatometer’s 9 correlations changing by a mean of 0.020.
In other words, this data estimation process was very successful.
**********
Armed with complete data, I used factor analysis[3] to calculate a single “film quality” score. The results were nearly identical to those from the previous post: using all 10 variables yields two factors accounting for 68.1% (PQ) and 26.0% (Public Awareness) of the total variance, while removing year of release, runtime and numbers of raters yields one factor (PQ) accounting for an astonishing 92.6% of the total variance: IMDb users, RT critics, RT fans and Maltin rate movies in remarkably similar ways. Table 3 presents the factor loadings (correlations with underlying dimension being assessed) and score coefficients used to generate a single PQ score.
Table 3: Factor Analysis of Film Quality Measures, Two Iterations
| Variable | All 10 Variables 68.2% | “Rating” Variables Only 92.6% | ||
| Loadings | Coefficients | Loadings | Coefficients | |
| Year of Release | -0.172 | 0.013 | ||
| Length (mins.) | 0.248 | 0.017 | ||
| IMDb Score | 0.935 | 0.227 | 0.914 | 0.104 |
| IMDb Raters | 0.425 | 0.035 | ||
| Tomatometer | 0.844 | 0.099 | 0.869 | 0.142 |
| RT Critic Average | 0.921 | 0.311 | 0.932 | 0.339 |
| RT Critics | 0.391 | 0.059 | ||
| RT Audience Score | 0.919 | 0.194 | 0.923 | 0.253 |
| RT Audience Average | 0.903 | 0.171 | 0.903 | 0.203 |
| Maltin Stars | 0.679 | 0.038 | 0.691 | 0.037 |
From these coefficients, Stata[4] calculated two PQ scores – PQAll and PQRating – for each film. Think of these values as SD above or below 9 to 5. They are correlated a whopping 0.992, though they do have subtle differences, as Table 4 reveals:
Table 4: 30 Highest Rated Films I Have Seen Multiple Times, Compared by PQ Score
| PQAll | PQScore |
| 30. The Apartment | 30. The Maltese Falcon (1941) |
| 29. Memento | 29. The Cabinet of Dr. Caligari |
| 28. No Country For Old Men | 28. Witness for the Prosecution |
| 27. Some Like It Hot | 27. L.A. Confidential |
| 26. Sherlock Jr. | 26. The Apartment |
| 25. Vertigo | 25. Vertigo |
| 24. The Wizard of Oz | 24. Annie Hall |
| 23. The Third Man | 23. Back to the Future |
| 22. L.A. Confidential | 22. The General |
| 21. The Avengers | 21. Kind Hearts and Coronets |
| 20. Double Indemnity | 20. The Wizard of Oz |
| 19. On the Waterfront | 19. Metropolis |
| 18. M (1931) | 18. Some Like It Hot |
| 17. Metropolis | 17. North By Northwest |
| 16. North By Northwest | 16. Star Wars Episode IV: A New Hope |
| 15. Chinatown | 15. Double Indemnity |
| 14. Back to the Future | 14. Chinatown |
| 13. Sunset Boulevard | 13. The Third Man |
| 12. Rear Window | 12. On the Waterfront |
| 11. Citizen Kane | 11. Rear Window |
| 10. It’s A Wonderful Life | 10. It’s a Wonderful Life |
| 9. Psycho | 9. M (1931) |
| 8. Indiana Jones and the Raiders of the Lost Ark | 8. Psycho |
| 7. Star Wars Episode IV: A New Hope | 7. Citizen Kane |
| 6. Casablanca | 6. Indiana Jones and the Raiders of the Lost Ark |
| 5. Inception | 5. Sunset Boulevard |
| 4. 12 Angry Men | 4. Sherlock Jr. |
| 3. Avengers: Endgame | 3. Pulp Fiction |
| 2. Pulp Fiction | 2. Casablanca |
| 1. The Dark Knight | 1. 12 Angry Men |
Just 24 films appear on both lists (including L.A. Confidential, my favorite movie). Six newer (median 2009), longer (146.5 minutes), more-oft-rated films (1,214,741 IMDb raters, 352 RT critics) films in the PQAll Top 30 are replaced by six older (1945), shorter (96.5 minutes), slightly-less-oft-rated films (131,469; 58.5). Only six films – It’s A Wonderful Life, Psycho, Indiana Jones and the Raiders of the Lost Ark, Casablanca, 12 Angry Men and Pulp Fiction – rank in the top 10 on both PQ scores.
Still, if I were to choose a set of recent films likeliest still to be highly regarded a few decades from now, Memento, No Country for Old Men, The Avengers, Inception, Avengers: Endgame and The Dark Knight are an excellent starting point. Three of them, along with The Prestige and Batman Begins, were directed by Christopher Nolan – perhaps the best director of our time.
While this is a very impressive list of films – I was even more blown away by 12 Angry Men the second time – it is based ONLY on films I have seen multiple times. It excludes highly-regarded films (per the 20 top-ranked films on IMDb) I have only seen once: The Shawshank Redemption, first two Godfather films, Fight Club, Forrest Gump, Star Wars Episode V: The Empire Strikes Back, The Matrix, Goodfellas, One Flew Over the Cuckoo’s Nest and Se7en. And then there are movies I have not seen at all: Schindler’s List; the Lord of the Rings trilogy; The Good, the Bad and the Ugly; and Seven Samurai.
In fact, I have only seen four of the top 20 IMDb-rated movies multiple times: The Dark Knight, 12 Angry Men, Pulp Fiction and Inception. This reflects my personal taste in movies: older, noir-tinged, mysteries and comedies rather than more contemporary fantasy, war-based or western films; whereas the median year of release of the top 250 films by PQScore is 1971.5, respectively, that of the 250 films by IMDb score is 1994 – with 51 being released in 2011 or later, a strong indication of recency bias in the IMDb score data.
Another way to consider my particular taste in movies is to examine the distribution of year of release (Figure 3). There are two distinct peaks –1946-50 (n=73, 11.4%) and 1982-98 (n=256, 40.1%). The former period roughly corresponds to the pinnacle of classic film noir, while the latter is my primary movie-attending period – ages 15 to 32. Indeed, 119 (18.7%) of these films are classic-era films noir, released between 1940 (Stranger on the Third Floor) and 1958 (Touch of Evil) with average PQAll and PQScore of 0.03 and 0.08, respectively (relative to 0, overall). This excludes 18 films directed by Alfred Hitchcock, 10 of which are widely considered film noir, with average PQAll and PQScore of 0.92and 0.94, respectively. Finally, there are 49 films released before 1960 – not film noir or Hitchcock, not Charlie Chan (n=23; both -0.66), starring The Marx Brothers (n=9; 0.38, 0.48) – with averages of 0.92 and 0.97, respectively. The other 395 films (61.9%) – excluding the 25 directed by Woody Allen[5] (n=25; 0.26, 0.30) – have averages of -0.15 and -0.18, respectively. These values are broadly similar to those from the previous post, excepting the addition of “Charlie Chan.”
Figure 3: Distribution of Year of Release is Bimodal
Even more instructive is to compare my favorite film by time period (>5 films) – or, at least, my best guess absent a formal assessment – to the film with the highest PQAll and PQScore from that period.
1920 to 1930
Number films = 9
Average PQAll = 1.12
Average PQScore = 1.20
Top film: Metropolis (1.61, 1.53)
Personal favorite: The Phantom of the Opera (0.76, 0.84), albeit barely
Comment: Phantom has the lowest PQ scores of the nine films released in the 1920s I have seen multiple times, revealing their overall quality.
1931
Number films = 5
Average PQAll = 0.65
Average PQScore = 0.74
Top film: M (1.61,1.65)
Personal favorite: M
Comment: I have seen at least five other films released in 1931, making it a key year in my personal fandom
1932 to 1934
Number films = 5
Average PQAll = 0.72
Average PQScore = 0.83
Top film: The Thin Man (1.32,1 .43)
Personal favorite: The Thin Man
1935
Number films = 5
Average PQAll = 0.27
Average PQScore = 0.34
Top film: A Night at the Opera (1.23, 1.30)
Personal favorite: A Night at the Opera
1936 to 1939
Number films = 29
Average PQAll = -0.18
Average PQScore = -0.17
Top film: The Wizard of Oz (1.57, 1.50)
Personal favorites: After the Thin Man (0.83, 0.94), Charlie Chan at Treasure Island (-0.18, -0.15), Charlie Chan in Reno (-0.62, -0.64)
Comment: For me, the 1930s combine genuinely great films – eight from the Marx Brothers – with 16 Charlie Chan films.
1940
Number films = 10
Average PQAll = 0.19
Average PQScore = 0.24
Top film: Rebecca (1.43, 1.41)
Personal favorite: Foreign Correspondent (0.66, 0.72), though Rebecca and His Girl Friday (1.28, 1.35) are wicked close
Comment: Hitchcock made his first Hollywood films in 1940, and they were knockouts…though the best was yet to come.
1941
Number films = 11
Average PQAll = 0.19
Average PQScore = 0.23
Top film: Citizen Kane (1.75, 1.66)
Personal favorite: The Maltese Falcon (1.38, 1.44)
Comment: Film noir bursts on the scene with a bang.
1942
Number films = 9
Average PQAll = 0.04
Average PQScore = 0.07
Top film: Casablanca (1.88, 1.77)
Personal favorite: All Through the Night (0.15, 0.26)
Comment: …as does Humphrey Bogart.
1943 to 1944
Number films = 10
Average PQAll = 0.64
Average PQScore = 0.70
Top film: Double Indemnity (1.61, 1.59)
Personal favorite: Laura (1.25, 1.31)
Comment: 1944 is the best year for film quality (0.74, 0.79) since 1931
1945
Number films = 7
Average PQAll = -0.25
Average PQScore = -0.20
Top film: Scarlet Street (0.82, 0.91)
Personal favorite: Spellbound (0.40, 0.43)
Comment: Followed by a mediocre 1945.
1946
Number films = 17
Average PQAll = 0.01
Average PQScore = 0.05
Top film = It’s A Wonderful Life (1.77, 1.65)
Personal favorite: Deadline at Dawn (-0.56, -0.50)
Comment: This is the first large disconnect between the “best” film released in a year and my favorite film from that year.
1947
Number films = 19
Average PQAll = -0.18
Average PQScore = -0.22
Top film: Out of the Past (1.77, 1.65)
Personal favorite: Out of the Past
1948
Number films = 13
Average PQAll = 0.30
Average PQScore = 0.38
Top film: Rope (0.91, 0.94)
Personal favorites: Call Northside 777 (0.09, 0.09) and The Naked City (0.39, 0.43)
1949
Number films = 11
Average PQAll = 0.26
Average PQScore = 0.30
Top film = The Third Man (1.58, 1.62)
Personal favorite: Impact (-0.60, -0.65)
1950
Number films = 13
Average PQAll = 0.08
Average PQScore = 0.12
Top film: Sunset Boulevard (1.70, 1.69)
Personal favorite: Where the Sidewalk Ends (0.59, 0.71)
Comment: Classic film noir and Hitchcock yielded some of the best films ever made.
1951
Number films = 8
Average PQAll = 0.20
Average PQScore = 0.28
Top film: Strangers on a Train (1.27, 1.31)
Personal favorite: The Enforcer (-0.19, -0.15)
1952
Number films = 6
Average PQAll = -0.05
Average PQScore = -0.01
Top film: The Narrow Margin (0.72, 0.85)
Personal favorite: Kansas City Confidential (-0.05, -0.02)
Comment: This is the first time the best film I have seen multiple times released that year is far from the best film released that year. That honor goes to Singin’ in the Rain, a movie I have yet to see in its entirety.
1953
Number films = 7
Average PQAll = -0.02
Average PQScore = 0.02
Top film: The Big Heat (1.04, 1.12)
Personal favorite: 99 River Street (0.35, 0.47)
1954
Number films = 7
Average PQAll = 0.40
Average PQScore = 0.38
Top film: Rear Window (1.72, 1.64)
Personal favorite: Dial M For Murder (0.88, 0.84), edging out Rear Window
Comment: I have yet to see Seven Samurai…and Hitchcock has hit his absolute peak.
1955
Number films = 7
Average PQAll = 0.77
Average PQScore = 0.85
Top films: Diabolique (1.25, 1.27) and Rififi (1.24, 1.27)
Personal favorite: Muerte de un ciclista (Death of a Cyclist) (0.70, 0.82)
Comment: For the first time since 1920-31, foreign films dominate – with Germany replaced by France and Spain. It also marks the shift the overseas shift of film noir, presaging La Nouvelle Vague.
1956 to 1959
Number films = 15
Average PQAll = 0.94
Average PQScore = 0.94
Top film: 12 Angry Men (1.95, 1.80)
Personal favorite: 12 Angry Men
1960-69
Number films = 26
Average PQAll = 0.31
Average PQScore = 0.33
Top film: Psycho (1.79, 1.66)
Personal favorite: The Apartment (1.49, 1.44), though Psycho is close.
Comment: I have not seen multiple times any films released in 1961, 1969 or 1970. And most of the 1960s is a cinematic wasteland for me…though I may have seen Dr. Strangelove, or How I Stopped Worrying and Learned to Love the Bomb more than once.
1970 to 1972
Number films = 6
Average PQAll = 0.36
Average PQScore = 0.39
Top film = Sleuth (1.17, 1.19)
Personal favorites: What’s Up, Doc? (0.70, 0.76) and Willie Wonka and the Chocolate Factory (0.79, 0.79)
Comment: The Godfather – which I have only seen once – is the best film released in 1972. Some would argue…ever.
1973
Number films = 8
Average PQAll = 0.44
Average PQScore = 0.46
Top films: Paper Moon (1.32, 1.36) and The Sting (1.37, 1.31)
Personal favorite: Charley Varrick (0.45, 0,49) edges The Sting and American Graffiti (0.83, 0.90).
1974 to 1975
Number films = 11
Average PQAll = 0.47
Average PQScore = 0.51
Top film: Chinatown (1.66, 1.61)
Personal favorite: Murder on the Orient Express (0.41, 0.43)
Comment: Chinatown is a masterpiece; The Godfather: Part 2 (only seen once) is considered better.
1976 to 1977
Number films = 10
Average PQAll = 0.19
Average PQScore = 0.20
Top film: Star Wars Episode IV: A New Hope (1.87, 1.57)
Personal favorites: Murder by Death (-0.06, -0.05) and The Seven-Per-Cent Solution (-0.46, -0.39)
1978
Number films = 11
Average PQAll = -0.64
Average PQScore = -0.61
Top film: Superman (0.81, 0.85)
Personal favorites: Death on the Nile (0.04, 0.03) and Thank God It’s Friday (-2.26, -2.24)
Comment: I genuinely do not understand why the charming Friday does get more love. And as solid as Superman is, The Deer Hunter (which I have not seen) is probably the best film released in 1978.
1979
Number films = 12
Average PQAll = 0.06
Average PQScore = 0.11
Top films: Being There (1.20, 1.20) and Manhattan (1.19, 1.21)
Personal favorites: Manhattan and The In-Laws (0.50, 0.61)
1980
Number films = 12
Average PQAll = -0.22
Average PQScore = -0.18
Top film: Airplane! (1.02, 1.06)
Personal favorite: Times Square (-0.60, -0.55)
Comment: I have only seen Empire Strikes Back once.
1981
Number films = 11
Average PQAll = 0.23
Average PQScore = 0.29
Top film: Indiana Jones and the Raiders of the Lost Ark
Personal favorite: Indiana Jones and the Raiders of the Lost Ark
1982
Number films = 17
Average PQAll = 0.06
Average PQScore = 0.09
Top film: Blade Runner (1.30, 1.14)
Personal favorites: Fast Times at Ridgemont High (0.13, 0.15) and Hammett (-0.60, -0.52)
Comment: Wow, Harrison Ford dominated movies from 1977 and 1983.
1983
Number films = 9
Average PQAll = -0.18
Average PQScore = -0.12
Top film: A Christmas Story (1.14, 1.15)
Personal favorite: Valley Girl (-0.22, -0.06)
1984
Number films = 19
Average PQAll = -0.34
Average PQScore = -0.30
Top film: This is Spinal Tap (1.22, 1.26)
Personal favorite: The Cotton Club (-0.67, -0.66)
Comment: We are now squarely in the age of mediocre films I saw in the theater as a teenager/young adult then chose to watch again. And maybe I need to see Amadeus again.
1985
Number films = 14
Average PQAll = 0.22
Average PQScore = 0.24
Top film: Back to the Future (1.70, 1.48)
Personal favorite: Back to the Future, though The Sure Thing is close (0.09, 0.18)
1986
Number films = 17
Average PQAll = -0.16
Average PQScore = -0.14
Top films: Hannah and Her Sisters (1.11, 1.12) and Blue Velvet (1.07, 1.08)
Personal favorite: While Legal Eagles (-1.73, -1.83) is a top guilty pleasure, I may like Hannah more.
Comment: Yes, I have never seen Aliens.
1987
Number films = 17
Average PQAll = -0.33
Average PQScore = -0.32
Top film: Wings of Desire (1.39, 1.41)
Personal favorite: House of Games (0.57, 0.67)
1988
Number films = 13
Average PQAll = -0.43
Average PQScore = -0.41
Top film: Die Hard (1.48, 1.33)
Personal favorite: Who Framed Roger Rabbit (0.97, 0.97)
1989
Number films = 13
Average PQAll = -0.49
Average PQScore = -0.49
Top film: Crimes and Misdemeanors (0.98, 1.00)
Personal favorite: Forced to choose from a lot of meh, I pick The Big Picture (-0.84, -0.77) for its charming cast.
1990
Number films = 13
Average PQAll = -0.32
Average PQScore = -0.29
Top film: Metropolitan (0.58, 0.66)
Personal favorite: Metropolitan
Comment: The one film released in 1990 in the IMDb Top 250 is Goodfellas, which I look forward to rewatching. [Ed. note: I neglected to add Awakenings, which I have seen twice, and which tops Metropolitan among films I have seen multiple times – though the latter is still my personal favorite among these group.]
1991
Number films = 20
Average PQAll = -0.70
Average PQScore = -0.73
Top film: JFK (0.99, 0.85)
Personal favorite: If I have to pick one from this meh collection – Dead Again (0.15, 0.23)
Comment: I should watch The Silence of the Lambs again.
1992
Number films = 16
Average PQAll = -0.63
Average PQScore = -0.64
Top film: The Player (0.94, 0.98)
Personal favorite: The Public Eye (-0.99, -1.01)
Comment: I should watch Reservoir Dogs again – as we hit rock bottom in the early 1990s.
1993
Number films = 13
Average PQAll = -0.85
Average PQScore = -0.85
Top film: The Fugitive (1.01, 0.95)
Personal favorite: Manhattan Murder Mystery (0.34, 0.40)
Comment: I spoke too soon…yeesh. Perhaps once I finally see Schindler’s List. [Ed. note: A few days after posting this, I watched Dazed and Confused for the second time, and I think it replaces both The Fugitive and Manhattan Murder Mystery – making it both the “best” film and my favorite film release in 1993 I have seen multiple times. I have also seen Jurassic Park only once.]
1994
Number films = 18
Average PQAll = -0.32
Average PQScore = -0.35
Top film: Pulp Fiction (2.16, 1.72)
Personal favorite: The Shadow (-1.63, -1.74) – easily the widest gap between “best” and “favorite”
Comment: Do not be fooled by these data. Hollywood has now entered an absolute golden age, rivaling its best years. Remember, I have only seen Shawshank Redemption, Forrest Gump and the phenomenal Léon: The Professional once. And I have never seen The Lion King. These four movies and Pulp Fiction are in the top 40 films by IMDb score.
1995
Number films = 12
Average PQAll = -0.28
Average PQScore = -0.34
Top film: The Usual Suspects (2.16, 1.72)
Personal favorite: The Usual Suspects
Comment: Eight of the IMDb Top 250 were released in 1995.
1996
Number films = 15
Average PQAll = -0.36
Average PQScore = -0.36
Top film: Fargo (1.42, 1.31)
Personal favorite: Big Night (0.69, 0.78), probably.
1997
Number films = 9
Average PQAll = 0.24
Average PQScore = 0.20
Top film: L.A. Confidential (1.59, 1.44)
Personal favorite: L.A. Confidential
Comment: I am still upset L.A. Confidential did not win the Academy Award for Best Picture…and Titanic is not even in the IMDb Top 250; L.A. Confidential only ranks 5th.
1998
Number films = 16
Average PQAll = -0.24
Average PQScore = -0.26
Top film: Rushmore (1.08, 1.07)
Personal favorites: Dark City (0.45, 0.37) and Pleasantville (0.54, 0.46)
Comment: I have zero interest in Saving Private Ryan – in fact, Steven Spielberg does almost nothing for me. There, I said it. And I just realized I left The Big Lebowski off my list – though it would not rank much higher than Rushmore.
1999
Number films = 14
Average PQAll = -0.40
Average PQScore = -0.51
Top film: Toy Story 2 (1.34, 1.19)
Personal favorites: Cradle Will Rock (-0.24, -0.29) and Mystery Men (-0.91, -0.96)
Comment: I may watch Fight Club and The Matrix again at some point.
2000 to 2001
Number films = 12
Average PQAll = 0.35
Average PQScore = 0.21
Top film: Memento (1.52, 1.19)
Personal favorite: Mulholland Drive (1.03, 0.79)
2002-09
Number films = 20
Average PQAll = 0.19
Average PQScore = -0.14
Top film: The Dark Knight (2.19, 1.39)
Personal favorite: The Dark Knight
Comment: For the first time, the difference in scores reflects the bias in PQAll toward longer, more recent films with many raters.
2010-19
Number films = 12
Average PQAll = 1.09
Average PQScore = 0.72
Top film: Avengers: Endgame (2.03, 1.30) and Inception (1.95, 1.17)
Personal favorite: Predestination (0.28, 0.10)
Comment: We come full circle: as in the 1920s, the handful of films released in the 2010s I have seen multiple times are generally well-regarded. And my favorite (sorry, Endgame, Hugo, Doctor Strange) is again least well-regarded, but still better than average.
**********
Figure 4 reinforces the year-by-year analysis. It shows early and late peaks: one for the 1920s because I carefully chose the best films to watch and rewatch, and one for the 2010s, because I stopped seeing movies in theatres first, choosing the best films to watch (OK, and all 23 Marvel Cinematic Universe films) then watch again. Moreover, average PQScore is higher than average PQAll until about 1940 – when it basically draws even until the early 1990s – because the latter elevates longer, recent, heavily-reviewed films. Around 2000, average PQAll pulls much further ahead for the same reason.
After a sharp decline in both values through the 1930s (excepting a slight uptick in 1931) – reflecting the Fox Charlie Chan films I love more than most – the emergence of film noir around 1940 pulls values up again. They generally stay above 0 through the early 1950s, peaking sharply in 1944. A preponderance of exceptional Hitchcock and foreign films sends scores skyrocketing in the late 1950s, with a lower peak from the 1960s into the mid-1970s: once again a lack of films elevates scores.
Figure 4: Average PQAll and PQScores by Year of Release/Midpoint of Range of Year of Release, 1920-2019
Almost by definition, I first saw any film released through the mid-1970s, not in a movie theater, but on television, through one of Yale’s films societies or through a rental/streaming service. This will inevitably bias toward “better” films. That changed in the late 1970s, when I began regularly seeing films for the first time in a movie theater – resulting in a much wider range of quality. The steep decline in scores in 1978 shows that, as does the zig-zagging around 0 through the mid-1980s. I cannot really explain why scores plummet in the late 1980s and early 1990s, though there is evidence overall film quality was much lower in those year.
The sharp spike up in 1994 masks an even more dramatic increase in overall film quality. I have long thought the peace and prosperity of the last six years of the Clinton Administration yielded a new golden age in American cinema; perhaps studio executives felt freer to experiment with original screenplays. I also though that changed after the 9-11, but…maybe not. Still, I suspect the preponderance of post-2000 films in the IMDb Top 250 reflects recency bias more than actual quality.
Basically, while I consider myself a cinephile, like everyone who watches movies my tastes range from genuine works of art to the guiltiest of guilty pleasures. And that is how it is supposed to be – we like what we like, not what we are “supposed” to like. Still, whether I am simply more honest in admitting how much I genuinely like the Fox Charlie Chan films, and films like Deadline at Dawn, Impact, The Seven-Per-Cent Solution, Thank God It’s Friday, Times Square, Hammett, The Cotton Club, Legal Eagles, The Big Picture, The Public Eye, The Shadow and Mystery Men remains to be seen.
In the meantime, I just shared Cat People (1942) with my wife Nell – who quite enjoyed it – so I need to update my Excel workbook. Again.
Until next time…please wear a mask as necessary to protect yourself and others – and if you have not already done so, get vaccinated against COVID-19! And if you like what you read on this website, please consider making a donation. Thank you.
[1] New York, NY: New American Library
[2] StataCorp. 2005. Stata Statistical Software: Release 9. College Station, TX: StataCorp LP.
[3] Specifically: factor analysis, principal factors, varimax rotation, forcing one or two factors, depending on input variables.
[4] Using “Predict” command in Stata
[5] Despite my ambivalence about Allen as a human being, I still love many of his films.
Just Bear With Me... 
3 thoughts on “Measuring Film Quality: Revisiting “Guilty Pleasures””