Organizing by themes I: American politics

This site benefits/suffers/both from consisting of posts about a wide range of topics, all linked under the amorphous heading “data-driven storytelling.”

In an attempt to impose some coherent structure, I am organizing related posts both chronologically and thematically.

Given that I have multiple degrees in political science, with an emphasis on American politics, it is not surprising that I have written a few dozen posts in that field…and that is where I begin.

I Voted sticker

**********

I started by writing about the 2016 elections, many based on my own state-partisanship metric (which I validate here).

The absurdity of the Democratic “blue wall” in the Electoral College

Hillary Clinton’s performance in five key states (IA, MI, OH, PA, WI)

Why Democrats should look to the south (east and west)

How having (or not) a college degree impacted voting

An alternative argument about gerrymandering

An early foray into what I call “Clinton derangement”

The only statistic from 2016 that really matters

**********

Here are a few posts about presidential polling (before FiveThirtyEight jumped on the bandwagon)…

Be careful interpreting President Trump’s approval polls

…and the 2017 special election in Georgia’s 6th Congressional District (GA-6)

Ossoff and the future of the Democratic Party

Using GA-6 polls to discuss statistical significance testing (spoiler: I am not a fan)

**********

And then I started looking ahead to 2018…first to control of the United States House of Representatives (“House”). Note that posts are often cross-generic…

An alternative argument about gerrymandering

The impact of voting to repeal (and not replace) Obamacare (May 2017)

I debut my simple forecast model (June 2017)

Making more points about polls and probability

A March 2018 update

A followup March 2018 update (after which I stopped writing about the 2018 House elections)

…then the United States Senate

The view from May 2017

What it meant that the Senate voted NOT to repeal Obamacare in July 2017

The view from December 2017

…and, finally, races for governor in 2017 AND 2018.

The view from June 2017

A tangentially-related post may be found here.

**********

After Labor Day 2018, I developed models (based on “fundamentals” and polls) to “forecast” the Senate elections…

September 4

September 13

October 23

…and those for governor (the October 23 post addressed both sets of races)

September 16

These culminated in…

My Election Day cheat sheet

And my own assessment of how I did (spoiler: not half bad)

Speaking of assessments, I took a long look at my partisan lean measure here.

And I carefully examined some polling aggregation assumptions here.

**********

Beginning in April 2019, I turned my attention to the 2020 elections.

First came a wicked early look at the relative standings of the dozens of women and men actually or potentially seeking the 2020 Democratic presidential nomination:

April 2019

Then came a wicked early look at the 2020 presidential election itself.

April 2019

And, of course, a wicked early look at races for Senate (2020) and governor (2019-20).

With a post-Labor-Day update.

With the first of regular updates to both the 2020 Democratic presidential nomination and the 2020 presidential election in May 2019

This post both set up the first Democratic debates and had good news for Democrats looking ahead to 2020.

This post set up the second Democratic debates and drew some conclusions about who “won” and “lost” the first debates.

This post updated the data for August 2019 and drew some conclusions about who “won” and “lost” the second debates.

Ditto for September 2019, October 2019, November 2019,  December 2019, January 2020

Once voting commenced in the 2020 Democratic presidentil nomination process, I wrote posts specific to the

As for the 2020 general election:

And two assessment of Emerson College polls (one, two).

**********

Finally, there are other politics posts that defy easy categorization.

I indulged in some speculative alternative history about the presidential elections of 1948 and 2000.

I delineated issue differences between Democrats and Republicans.

I got a bit personal here and here, concluding with the fact that, despite overlapping in the same residential college at Yale for two years, I did NOT know Associate Justice Brett Kavanagh at all.

I argued for the abolition of the Electoral College.

Until next time…

Bipartisanship as patriotism

I started quietly screaming here.

But my deep revulsion for what the United States government, my government, the government elegantly outlined in our founding documents, is doing along our southern border (not the northern border with majority-northern-European Canada, mind you) boiled over the other night in this (annotated) 1,000+-word reply to a similar cri de coeur on the Bone and Silver blog.

The US faces an epistemological crisis. Some 20-25% of the population–primarily rural white Protestant men with at most a high school diploma (culturally conservative, isolationist, economically populist)–has been conditioned by right-wing propaganda (Fox News, talk radio mostly) for 30+ years to believe that all of their problems are caused by a long list of “others”: blacks (dangerous criminals), Spanish-speaking immigrants (drug-lord rapists and murderers who want your jobs), Muslims (terrorists), LGBQT folks (out to destroy your families), the mass media (lying to you), liberals (wimpy snowflakes who hate you and your values and *your* country) and the globalist-coastal elites (sending *your* jobs and country overseas, or something).

[Eds. note: I have no idea how large this segment of the population is. Trump’s 2016 share of the voting-age population was 25.0%, according to data from here and here. While not all Trump voters fit this characterization, an identical 25% (on average) support Trump’s recent immigration actions. And about 24% of American adults solely get their news from Fox News. The overlap between these groups is probably quite large, though well below 100%. Still, even if the percentage is only half of my upper limit—12.5%–that is still 1 in 8 Americans over the age of 18.] 

The crisis is that these Americans literally live in a different reality, with different news sources and accepted truths. This self-contained echo chamber is the only way they can sustain their paranoid grievances. And what they most fear is not loss of economic status but loss of racial/cultural status. They see an encroaching diverse modernity in which they have little-to-no status, which existentially terrifies them.

And so they cultishly follow an autocrat who echoes and validates their worst fears:  Mexicans and Muslims and transgendered folks and black athletes and liberals and Democrats and the media and China and our allies (Canada? Really?) are out to get *them*.

They are so deep in this twisted (yet infinitely self-justified) worldview that they no longer see these “others” as human beings, at some primitive level. *They* are animals who will “infest” (in 45’s words) THEIR country and destroy THEIR way of life. 

Yeah, you say, but they are outnumbered at least 3-1, so why is this happening?

This 20-25% of the population has an outsized influence on the Republican Party (which has cynically nurtured their paranoia for political gain since Nixon was first elected president in 1968), particularly which Republicans get nominated—and especially since the election of an urbane black man as president in 2008. That was a bridge too far for them, and for the Republican Party, who (to prevent losing nominations to further-right-wing candidates) vowed absolute opposition to him. They are also geographically dispersed across enough districts to elect enough like-minded Republicans to effectively control a majority of state houses and the United States House of Representatives. And, in a 17-person field, they coalesced around Trump early enough to allow him to win the nomination, sweeping aside an establishment that could not (or would not) coalesce around a more “mainstream” alternative (not that their choices were all that impressive). Once the Democrats nominated the equally-flawed Hillary Clinton, after Democrats had controlled the White House for 8 years…well, he still only won by 77,000 votes in three states (while losing the popular vote by 2.1 percentage points—the Electoral College’s Republican advantage at work again).

The thing is, 45’s policy advisors–including the all-but-Nazi Stephen Miller–truly think that they beat Clinton not because she was a bad candidate at the wrong time, but because they mistakenly believe that most of the country is as right-wing nationalist/racist as they are. Here, they are flat wrong, but for arcane structural reasons, it may still take a tidal wave of Democratic votes to wrest back the House this November (the Senate will be tougher, but I am optimistic). 

And as with any tribalist cult, they make up in passion and cunning what they lack in numbers, including voting at higher rates, while using every trick to maximize their electoral advantage (less through gerrymandering than through suppression). They do this because they legitimately see the “not-them” as Manichean enemies who must be stopped at all costs. For them, ends justify cruel, immoral and, yes, anti-democratic means: when push comes to shove, safety/security generally trumps (pun intended) liberal democracy. 

The thing is, though, even if Democrats win back the House (likely) and the Senate (30% chance?) and a bunch of state houses…actually, many good things will happen (if only by preventing more bad things from happening). But the crisis will still exist. This squeaky-wheel minority will, if anything, feel more aggrieved and more isolated and more desperate to fight inexorable change. And Fox News and Rush Limbaugh and Alex Jones and the National Enquirer and Breitbart will continue to echo and amplify their increasingly-distorted reality, not only because it serves their own interests (and bottom-lines) to do so–they also genuinely fear the consequences of suddenly backing off decades of crazy-stroking. 

So how do we fix this? How do we get a sprawling, impossibly-diverse nation of nearly 400 million people back on the same “we are all in this together” page (begging the question whether, besides WWII, we ever were)? How do we get these reality-denying folks to accept the reality of climate change, the trade-offs between secure borders and nurturing compassion, the tragic consequences of an overly-gun-permissive society (the unique Constitutional protection afforded guns has morphed into Constitutional protection of THEIR way of life—restricting the former is a direct assault on the latter), the value of expertise, the benefits of a multi-cultural/multi-ethnic society (a wider talent pool, if nothing else), and so forth?

I have absolutely no idea.

But as I see one California couple raise nearly $15 million almost overnight on Facebook to provide legal services for these newly-detained immigrants and their lost children, as I see more and more Republicans abandoning/staring down their party (thank you, Massachusetts Governor Charlie Baker), as I see the mainstream media absolutely refusing to back down from their Constitutionally-protected duty to investigate and report and expose, as I see Robert Mueller—a lifelong Republicandiligently pursuing his own investigations, as I watch previously apathetic citizens taking to the streets in protest…I have hope that the “sensible” (if not always ideologically-unified) 75+% will regain the “values” upper-hand and restore everything I have always loved about my country. 

The aggrieved minority may never accept what we understand as reality, because it is too existentially painful. But they are still my fellow Americans, and I must share our nation with them, just as they have to share it with folks like me. All I can do is continue to call out their nonsense in the clearest possible terms in the perhaps-naive hope that enough of them will eventually snap out of it.

Otherwise…we may simply have to wait as their numbers shrink even further, as the demographers insist will happen. 

Do not give up on this country…we ARE better than this.

Upon further reflection, though, I do have one practical suggestion, however, though it may not appeal to everyone: active bipartisanship.

It is telling in this regard that my second-ever post presented my bipartisan bona fides. My goal was to insulate myself against criticism (yet to materialize) that my liberal Democratic views biased my political and cultural data analyses. My meticulous sourcing also serves that purpose—allowing critical readers to fact-check my assertions and draw their own conclusion. In this, my academic roots clearly show: transparency in methods, data and sources.

But I think that post also stemmed from my hope that sufficient elected Republicans would stand up to the newly-elected President, thwarting his most anti-democratic impulses.

Shockingly few Republican elected officials, however, have done so. Yes, Republican Senators Susan Collins (Maine), John McCain (Arizona) and Lisa Murkowski (Alaska) voted NOT to repeal the Affordable Care Act. And Republican Senators Bob Corker (Tennessee) and Jeff Flake (Arizona), both of whom chose not to seek reelection in 2018, have at time publicly expressed deep reservations about President Trump.

But those moments have been few and far between. The reality is that Republicans, for all their protestations, have mostly voted for whatever President Trump has wanted. According to the FiveThirtyEight vote tracker, the median Republican United States Senator (51 currently serving) has voted with the President’s position a median 93.2% of the time, with 41 (80.4%) voting with his position at least 90% of the time; the “least” loyal Republican Senators were Rand Paul (Kentucky) and Collins, who still supported the President on at least 75% of votes. The obeisance was slightly higher for Republican members of the United States House of Representatives (US House; 235 currently serving who have cast at least one vote[1]): median support was 96.2%, with 193 (82.1%) voting with the President at least 90% of the time; the two least-loyal Republican House members have only voted with the President half of the time—Walter Jones (NC-3; 52.2%) and Justin Amash (MI-3; 53.0%). Curiously, the most vulnerable Republican House members, the 22 who represent congressional districts Clinton won in 2016, backed the President a median 97.0% of the time.

Instead, the few “profiles in courage” have come from state houses. Thirty-three states currently have Republican governors, with 16 having Democratic governors; Alaska Governor Bill Walker is an Independent.

Ohio Governor John Kasich famously challenged Trump from the (relative) left during the 2016 Republican presidential primaries and caucuses; he remains a vocal thorn in the President’s side. Three other Republican governors: Baker, Larry Hogan (Maryland), Phil Scott (Vermont)—remain enormously popular (68% approve/18% disapprove, on average) in states that are 24.1 percentage points more Democratic than the nation as a whole (using this calculation). Besides being genuinely likable, they remain popular by working—often in direct opposition to “their” President—closely with their states’ majority Democratic legislatures, carving out socially moderate-to-liberal and fiscally conservative positions.

Although I have lived in Massachusetts for most of the last 30 years, I never really followed Baker’s ascent, though I knew he was the chief Republican “up-and-comer” after his successful stint directing Harvard Pilgrim Health Care starting in 1999. In 2010, he was the Republican nominee against incumbent Democratic Governor Deval Patrick; Baker lost 48.4 to 42.0%.

charlie baker

A few months later, I was sitting in a Boston restaurant having lunch with my then-supervisor, when she nudged my arm. “Isn’t that himself?” she asked. I turned around to see Baker walk right near out table.That was when I realized how TALL he is (6’6”).

On August 25 of the previous year, Democratic Senator Edward M. Kennedy had died, after serving in the US Senate for almost 47 years. A special election to fill the seat through January 2013 was held on January 19, 2010. Democratic Attorney General Martha Coakley and little-known Republican State Senator Scott Brown easily won their primaries, and the prevailing wisdom was that Coakley would easily prevail against Brown. Instead, Brown upset Coakley 51.9 to 47.1%. (I drove through central Massachusetts with both daughters the weekend before the election, seeing no Coakley signs but quite a few Brown signs; uh-oh, I thought).

Four years later, with Patrick term-limited, Coakley was now the Democratic nominee for governor, seemingly a stronger candidate after her upset defeat. Baker was again the Republican gubernatorial nominee. And this time he won, 48.4 to 46.5%.

I did not vote for Baker in 2014 (just as I did not vote for Republican gubernatorial nominee William Weld in 1990 when he was, in many ways, more liberal than Democratic nominee Jon Silber—I now regret that vote). However, watching the debates between Coakley and Baker, I was struck by how much I LIKED Baker. Where Coakley was robotic and stiff, Baker was warm and engaging. His Harvard-educated brilliance shown through, but with an appealing everyman demeanor: he was clearly enjoying himself.

Because I think Coakley, with her flaws, would still have been a good governor, I do not regret my vote. But neither was I particularly upset that Baker won.

And since then, I have only grown to respect Baker more. He is more fiscally conservative than I would prefer, but his consistent willingness to call out Trump when necessary, well, trumps those positions.

I was wavering on voting for him this November (regardless of who the Democratic nominee is) until he forcefully “revoked his decision to send National Guard helicopters and personnel to the Southwestern border,” citing the inhumane treatment of children by the Trump Administration.

That did it: Nell and I will be voting to reelect Baker this fall, even as we joyfully vote for Democratic Senator Elizabeth Warren and our member of Congress, Joseph P. Kennedy III, also a Democrat.

Here is also why I will be voting for Baker in four+ months.

If I am calling on select Republicans to defy their President and work in a bipartisan fashion with Democrats, it would be massively hypocritical for me not to support a more-than-reasonable Republican who has done exactly that. Every time I cheer a former Republican speaking out against the President on MSNBC, I need to be able to match that gesture with one of my own.

Simply put, I cannot ask someone to do something—be actively bipartisan—without being willing to do the same thing myself.

Moreover, the only way to break down the tribalist partisanship that causes us to see persons with the wrong “label” as a mortal enemy is to elevate bipartisanship into an act of patriotism.

The stakes of the Cold War were so monumental that partisanship was supposed to stop at the water’s edge: there was to be no squabbling over matters of life and death. While that was not always true, particularly as the Vietnam War divided the Democratic Party and Democrats took President Ronald Reagan to task for his aggressively anti-Soviet Union posturing, that credo still serves as an excellent model for reimagining bipartisanship as patriotism.

Would I still vote for Baker if he were not heavily favored to win, meaning Nell’s and my votes will in no way be decisive? I do not know, to be honest. But were he not so effective AND anti-Trump, he would not be so popular, so the question kind of answers itself.

It is exceptionally difficult for lifelong partisans like me—this will only be the second time I vote Republican—even to consider opposing point of view (though it can be done), let alone voting for a candidate of the opposite party. But I firmly believe these actions are the best—maybe the only—ways to begin to solve our current epistemological crisis.

Until next time…

[1] 240 overall

An update on projected 2018 Democratic U. S. House seat gains

UPDATED Midnight EST, November 20, 2018. As of this writing, Democrats have netted 38 seats in the United States House of Representatives, with three races still to be called. Democrat Ben McAdams narrowly leads incumbent Republican Mia Love in Utah’s 4th Congressional District (CD), while Democrats trail narrowly in California’s 21st and Georgia’s 7th CD. For a fuller analysis of the 2018 midterm elections, see here. For more details on called and uncalled races, see here and here.

**********

With Democrat Conor Lamb’s narrow victory over Republican Rick Saccone in the March 13, 2018 special election to fill the United States House of Representatives (House) seat vacated by Republican Tim Murphy, there were 238 Republican-held seats, 193 Democratic-held seats and four open seats (2 Democratic, 2 Republican).

The two Democratic open seats (MI-13, NY-25) are likely to remain open until the entire House is up for election on November 6, 2018. In the last three presidential elections, the Democratic nominee won these Congressional districts (CD) by an average 67.4 and 18.3 percentage points (points), respectively, so it is highly likely both seats will be won by Democrats. The seat vacated by Republican Trent Franks (AZ-08) will be filled in a special election on April 24, while that of Republican Patrick Tiberi (OH-12) will be filled in a special election on August 7. The last three Republican presidential nominees won these two CDs by 22.7 and 10.2 points, respectively. The Arizona seat will likely remain Republican, though the Ohio seat could be close.

For now, however, let us assume that party control of these four CDs does not change. That would leave the partisan balance at 195 Democrats and 240 Republicans. This means that Democrats need to flip a net total of at least 23 seats to win a House majority after the 2018 midterm elections.

In a previous post, I listed projected Democratic net House seat gains based upon four models; I developed two of these models.

Briefly, I use the change in the margin by which Democrats won/lost the total nationwide vote for all 435 House seats (Democratic % – Republican %) from the previous Congressional elections (24 election, 1970-2016). In 2016, the Democrats lost the national House vote 47.6% to 48.7%, for a margin of -1.1 points[1]. The current FiveThirtyEight estimate is that Democrats leading in generic Congressional polls (“If the election was held today, would you vote for the Democratic or Republican candidate in your district?”) by 6.5 points (13.1% unsure/other parties).  If that were the actual election margin, that would equal a change of 6.5 – (-1.1) = 7.6 points.

In my “simple” ordinary least squares (OLS) regression model, I use only this value:

Estimated Democratic net seat gain = -1.63 + 3.11 * Change in Democratic margin

On average, every one percentage increase in the change in Democratic margin nets Democrats 3.1 additional House seats. Plugging a 7.6 point change in Democratic margin into the formula yields:

-1.63 + 3.11 * 7.6 = 22.01

This model has Democrats falling one seat shy of a House majority, even after winning the national House vote by 6.5 points—a result that should give believers in “one person, one vote” pause.

The 95% confidence interval (CI) around this estimate is 13.3 – 30.8, meaning we can say with 95% confidence that a 7.6 point shift in Democratic national House margin would result in a net gain of between 13 and 31 seats. This translates to a 41.2% chance the Democrats net at least 23 House seats in November 2018.

Again—that is when Democrats win the national House vote by 6.5 points. Take your pick of what to blame: incumbency advantage (Republicans gained a net 63 House seats in 2010 before the most recent round of redistricting), geographic self-sorting (Democrats in urban areas, Republicans in rural/small town areas, suburbs up for grabs) and partisan gerrymandering.

In a slightly more complex model, I account for whether the Congressional election coincided with a midterm or a presidential election:

If Midterm: Estimated Democratic net seat gain = -2.77 + 3.41 * Change in Democratic margin + 2.433

If Presidential: Estimated Democratic net seat gain = -2.77 + 2.08 * Change in Democratic margin

And here I make a confession.

In previous discussions of results from this complex model, I used an incorrect “intercept” term (-1.63 instead of -2.77), increasing estimated net Democratic House seat gains (by 1.14) and probabilities of House recapture.

Not a huge change, but I do strive for accuracy here.

Cutting to the chase, under the assumed 7.6 point change in national Democratic House margin, the complex model yields an estimated Democratic net gain of 25.6 seats (95% CI: -8.2 to 59.3[2]) with a 56.0% probability of regaining control of the House.

Here is Table 1 from my previous post updated to reflect the corrected “Berger 2” model and a required 23 net House seat gain for Democrats.

Table 1: Projections of 2018 Democratic net gains in House seats

Dem national House margin Abramson Brennan Berger 1 Berger 2
2% 19 5 8

(0%)

10

(12%)

4% 23 7 14

(1%)

17

(33%)

6% 27 13 20

(28%)

24

(52%)

8% 30 15 27

(78%)

31

(66%)

10% 34 21 33

(97%)

37

(75%)

11% 36 28 36

(99%)

41

(78%)

12% 38 31 39

(99+%)

44

(81%)

14% 42 41 45

(99+%)

51

(85%)

16% 46 56 52

 (100%)

58

 (88%)

And here is my updated graphic displaying the probability Democrats recapture the House given various changes in Democratic margin  (and dotted line indicating House control is 50-50):

Figure 1: Probability Democrats control U. S. House of Representatives after 2018 elections given change in Democratic margin from 2016

Democratic Probability 2018 House capture

You may ask why I assessed change in margin rather than actual margin. The two values are correlated r=0.55, meaning that there is a relatively strong, linear association between them (i.e., more often than not, when one increases/decreases the other increases/decreases), so the models should be broadly similar.

The outcome of interest to me is whether Democrats net at least 23 House seats they did not win in 2016. For that to happen, Democratic 2016 margins must increase from, say, 0 to -15 points to between +15 to 0 points. Republicans won 31 House seats in 2016 by ≤15 points, so a uniform swing (i.e. margin in every CD changes an identical amount) of 15 points toward the Democrats would net them eight more House seats than they need.

But let’s say the swing is not uniform. Perhaps the 195 Democratic-held seats see no change in margin in 2018, while the 240 Republican-held seats see a uniform shift of 15 points. That equates to an 8.3 point overall margin shift toward the Democrats, still netting them 31 seats (by comparison, my two models show an 8.6 point change in Democratic margin netting them 24-28 House seats).

Now, I could simply compare the actual Democratic margin in the national House vote to the actual number of seats won. The simple model is:

Estimated Democratic seats = 217.92 + 4.61 * Democratic margin

And the more complex model is:

If Midterm: Estimated Democratic seats = 211.29 + 4.18 * Democratic margin + 9.877

If Presidential: Estimated Democratic seats = 211.29 + 5.90 * Democratic margin

Plugging the current FiveThirtyEight estimate (Democrats+6.5) into either model yields an estimated 248 Democratic House seats—a net gain of 53 seats! That is a far more optimistic projection that those estimated using change in Democratic margin (22-26).

Moreover, if the Democrats and Republicans simply break even in 2018 (1.1 point increase from 2016), the “actual margin” models would project a bare Democratic House majority (218-221 seats), a net gain of 23-26 seats. By contrast, the “change in margin” models would project a net Democratic gain of only 2-3 seats under this assumption.

Statistically speaking, all four models are valid and account for more than 80% of the variance in the independent variable (net seat gain, seats won), a very high value for such simple models. Various indicators,[3] meanwhile, suggest that the simpler models are a better fit to the data than the complex models.

For now, though, I stick with the “change in margin” models as they align more with other projections (and a back-of-the-envelope seat-by-seat examination) and have, to me, a stronger conceptual underpinning.

That leaves one final test of the “change in margin” models.

One way to test the reliability (measuring the same underlying concept over repeated measurements) of OLS regression models is to remove, say, 1970 data, then rerun the regression. You would use the new model to estimate the value for the missing data point. Doing this for all (or a random subset) of data points yields illustrative comparisons between actual and estimated value, as shown in Table 2.

Table 2: Estimated (after removing that year’s data) and actual net Democratic change in House seats, 1970-2016

Year % Point Change in Dem  Margin Actual Net Dem House Seats Estimated –Actual Net Dem House Seats Model 1 Estimated –Actual Net Dem House Seats Model 2
1970 6.8 12 8.2 12.6
1972 -3.4 -13 0.8 3.6
1974 11.4 49 -17.9 -13.6
1976 -3.3 1 -13.6 -12.3
1978 -4.7 -15 -1.3 -1.5
1980 -6.0 -35 15.8 25.5
1982 9.1 27 -0.4 4.5
1984 -6.6 -16 -6.7 -0.6
1986 4.8 5 8.9 12.4
1988 -2.1 2 -10.6 -10.2
1990 0.1 7 -8.7 -7.6
1992 -2.8 -9 -1.4 0.5
1994 -12.0 -54 17.9 16.6
1996 7.1 3 19.1 12.4
1998 -1.2 4 -9.8 -9.2
2000 0.5 1 -1.1 -3.0
2002 -4.3 -8 -7.4 -7.8
2004 2.0 -2 6.9 3.8
2006 10.5 31 0.1 5.6
2008 2.6 23 -17.4 -22.8
2010 -17.2 -63 11.0 6.6
2012 7.9 7 17.6 9.9
2014 -7.1 -12 -12.8 -14.5
2016 4.7 6 7.5 1.2
Average, Raw Values 0.2 0.5
Average, Absolute Values 9.3 9.1
Average, Absolute Values, Midterms Only 8.7 9.4
Average, Absolute Values, Presidential Only 9.9 8.8

In the final two columns, a positive value means Democrats underperformed their estimated net House seat gain, while a negative values means they overperformed. Democrats underperformed and overperformed on both models 11 times each. In 1982 and 1992, the Democrats slightly overperformed on the simple model and slightly underperformed on the complex model.

The largest underperformances (≥10.0 in either model) were in 1980, 1994, 1996, 2010 and 2012; three were wave years (1980, 1994, 2010) in which Democrats averaged a net loss of 54 House seats. Similarly, the largest overperformances were in 1974, 1976, 1988, 2008 and 2014; strong waves occurred in 1974 and 2008 (Dems+36, on average) and 2014 (Dems-12). The pattern is not perfect, however, as the model was fairly close in the wave year of 2006 (Dems+31), missing by an average of three seats. Still, in 10 of 24 elections, at least one model missed the actual net change in Democratic House seats by ≥10 seats.

On average, estimates were spot on, as overperformance and underperformance essentially cancel out. However, when you examine the absolute value (difference in either direction) of differences, the models do worse, averaging +/- 9 seats. The slight differences between midterm and presidential election years make little practical difference.

Applying these average differences to the current projections (change from 2016 in Democratic national House margin of 7.6 points) yields

Berger 1: 22.0 +/- 9.3 = 12.7 to 31.3 net Democratic House seats

Berger 2: 25.6 +/- 9.1 = 16.5 to 34.7 net Democratic House seats

Put another way: Democrats would need to win the national House vote by 9.8 points in the simple model, and by 8.5 points in the complex model, for the lower end of these projected ranges to be 23 seats (what Democrats need to net to control the House after the 2018 midterm elections).

This is not how a representative democracy is supposed to work.

Until next time…

[1] Using data from Dave Liep’s indispensable Atlas of U.S. Presidential Elections

[2] This extremely wide CI results from using only 24 data points to estimate three parameters.

[3] Among them adjusted r-squared, residual sums of squares, degrees of freedom.

Doctor, validate thyself!

I recently wrote about my long-term fascination with American electoral geography, the way voting patterns are distributed across states, Congressional districts, counties and other areal units.

Pursuing this interest as an undergraduate political science major, I began to explore state-level presidential voting data. During my junior year, I created a large chart that ranked how states had voted in a series of recent presidential elections, from most to least Democratic, concluding with the 1984 presidential election (then the most recent one).

And I noticed that while Ronald Reagan, the incumbent Republican president, had absolutely walloped Democrat Walter Mondale in 1984, winning the popular vote by 18.2 percentage points (58.8-40.6%) and the Electoral College vote 525-13 (Mondale won only his home state of Minnesota [49.7-49.5%] and the District of Columbia [DC]), there were a few states Mondale lost by a much smaller margin than 18.2 percentage points: Massachusetts (-2.8 percentage points), Rhode Island (-3.6), Maryland (-5.5), Iowa (-7.4), Pennsylvania (-7.4), New York (-8.0) and Wisconsin (-9.2).

As usual, all presidential data are from Dave Leip’s indispensable Atlas of U.S. Presidential Elections.

Consider Pennsylvania, the state in which I was born. While the nation was voting for Reagan by 18.2 percentage points, Pennsylvania was voting for Reagan by “only” 7.4 percentage points (53.3-46.0%), a difference of 10.8 percentage points.

That is, Pennsylvania in 1984 was 10.8 percentage points MORE Democratic than the nation as a whole. Had Mondale lost by “only” 10 percentage points, he would (theoretically) have won Pennsylvania 25 electoral votes (EV), as well as those of Iowa (8), Maryland (10), Rhode Island (4) and Massachusetts (13)—an additional 60 EV.

And had Mondale lost by “only” 7.7 percentage points—as Democrat Michael Dukakis would to Republican George H. W. Bush in 1988—he would also have theoretically won the combined 53 EV of New York (36), Wisconsin (11) and West Virginia (6), boosting his total to 126 EV (better, but still 144 EV shy of the 270 needed to win the White House).

Still, that is close to the 112 EV Dukakis won in 1988.[1] As the purple-inked states on this beautiful hand-drawn map[2] show, Dukakis lost seven states (Illinois, Pennsylvania, Maryland, California, Vermont, Missouri, New Mexico) totaling 125 EV by smaller margins (2.1-5.0 percentage points; mean=3.3) than he did nationally. Had Dukakis lost the election by just 2.7 points, he would theoretically have won 237 EV, only 33 shy of the necessary 270.

1988 Presidential map

The conclusion I drew (no pun intended) was that the “relative partisan margin” of a state—how much more or less Democratic it was than the nation as a whole in a given election—was a useful way to think about electoral geography. Of course, other elections in the state (governor, United States Senate, United States House) are of interest as well, as Paul T. David observed in his Party Strength in the United States, 1872-1970; at one point, I even examined the partisan composition of state legislatures.

Good times.

Two decades later, despite having walked away from a doctoral program in political science, I was still interested in these questions, and I began to collect state-level presidential data again.

My primary goal was to get a sense of how EV’s would be distributed between the parties in the next presidential election (either 2008 or 2012) given a series of hypothetical national popular votes (e.g., Democrat wins nationally by 3 percentage points), essentially updating the exercises with 1984 and 1988 presidential election data I summarized earlier. I was particularly interested in whether the Democratic or the Republican presidential nominee would win more EV if the national vote were divided evenly between the two-major parties.

Having gathered these data, I set about constructing a measure of the relative partisanship of a state, intending to combine data from multiple elections to smooth out any idiosyncratic results.

For example, Democratic presidential nominees won Michigan by an average of 7.4 percentage points from 1992 through 2004, making the state an average 4.3 percentage points more Democratic than the nation. Democrat Barack Obama then won the Wolverine State by 16.4 percentage points in 2008 (9.2 percentage points better than he did nationally). In 2012 and 2016, meanwhile, the average margin in Michigan (with Republican Donald Trump winning by 0.2 percentage points in 2016) dropped to just 4.6 percentage points (only 1.6 percentage points more Democratic than the nation). A reasonable explanation (though not a conclusive one) for the Democratic spike in 2008 is the disproportionate impact of the 2007-08 recession on the automobile industry in Michigan, as voters took out their frustrations with term-limited President George W. Bush on 2008 Republican presidential nominee John McCain.

The questions then became

  1. How many years do I use?
  2. How, if at all, do I “weight” these elections?

My initial instinct was to use five years of data, with a weighting scheme of 1-2-3-4-5, meaning the least recent of the relative Democratic margins (D%-R% of total state vote minus D%-R% of total national vote) would be weighted 1/15 while the most recent one would be weighted 5/15, or 1/3.

This became my first “weighted relative Democratic margin” (W-RDM).

However, as I was also interested in assessing changes in relative state-level partisanship over time, using five elections meant that, prior to 2016, I only had four W-RDM values for a state—giving me only three election-to-election changes in W-RDM to examine[3].

I finally settled on three years in what I call my 3W-RDM[4] in order to minimize the fact that presidential and vice-presidential nominees tend to fare better, relative to their overall performance, in their home states. It is rare for one person to be on at least three consecutive presidential tickets (only two, George H W Bush, 1984-1992 and Gore, 1992-2000, of 21 total unique presidential and vice-presidential nominees, 1984-2012).

And that is the measure I have utilized in a series of posts (here, here, here; I do not specifically use 3W-RDM here, but the logic is the same).

As an example, here is how Nevada voted for president in 2004, 2008 and 2012:

             Year                State D% – R%                      National D% – R%              RDM

             2004                           -2.4                                               -2.5                         D+0.1

             2008                           12.5                                               7.3                         D+5.2

             2012                           6.7                                                 3.9                         D+2.8

The weighted average of the RDM values is (0.1 + 2*5.2 + 3*2.8)/6 = D+3.2. This was Nevada’s 3W-RDM prior to the 2016 election, so one would have expected that year’s Democratic nominee to do 3.2 percentage points better in Nevada than nationwide.

The 2016 Democratic presidential nominee, Hillary Clinton, won the national popular vote by 2.1 percentage points. So, my best estimate (based upon Nevada’s recent voting history) was that Clinton would win Nevada by 5.3 percentage points (2.1+3.2). This estimate was too optimistic, however, as she won Nevada by 2.4 percentage points, 2.9 percentage points lower than expected.

**********

Just bear with me while I briefly describe two other highly reputable approaches to calculating the relative partisan margin of a state (or other areal unit).

The Cook Political Report, the “independent, non-partisan newsletter that analyzes elections and campaigns for the US House of Representatives, US Senate, Governors and President as well as American political trends” has been essential reading for any serious student of American politics since its founding in 1984 by Charlie Cook, formerly “a staffer on Capitol Hill, a campaign consultant, a pollster, and a staff member for a political action committee.”

In 1997, Cook began to calculate the Partisan Voting Index (PVI) as a way to measure “how each [state or Congressional] district performs at the presidential level compared to the nation as a whole.”

The Cook PVI is simply the difference (state minus nation) between two averages:

  1. The average Democratic share of the state-level two-party vote in the previous two presidential elections
  2. The average Democratic share of the national two-party vote in the previous two presidential elections.

In 2008, Obama and McCain won 52.9% and 45.6%, of the national popular vote, respectively, splitting 98.5% of the total vote. Looking only at this two-party vote, Obama received 52.9/98.5 = 53.7% and McCain received 45.6/98.5=46.3%, meaning Obama beat McCain nationally by 7.4 percentage points in the two-party vote.

A similar calculation for 2012 (Obama 51.0%, Republican Mitt Romney 47.1%) shows that Obama beat Romney nationally in the two-party vote by 3.9 percentage points.

The average of 7.4 and 3.9 is 5.7.

In Nevada, meanwhile, overall Obama beat McCain 55.1-42.6%, and he beat Romney 52.4-45.7%; in the two-party vote, Obama won by margins of 12.8 (56.4-43.6%) and 6.8 (53.4-46.6%) percentage points.

The average of 12.8 and 6.8 is 9.8.

Subtracting 5.7 from 9.8 gives you 4.1, meaning that the PVI for Nevada going into 2016 was D+4.1, only a little more Democratic (D+3.2) than the 3W-RDM suggested.

The other approach is the “partisan lean” calculated by the data journalism website fivethirtyeight.com, a favorite of this blog.

It is even more straightforward than Cook PVI:

(RDM 2nd-most recent presidential + 3*RDM most recent presidential election)/4

Using Nevada again, we have already seen that in 2008 and 2012, Nevada voted 5.2 and 2.8 percentage points more Democratic than the nation; the 538 partisan lean (PL) formula gives you (5.2 +3*2.8)/4 = (5.2+8.4)/4=13.6/4=3.4.

Thus, Nevada’s 538 PL going into 2016 was D+3.4, broadly similar to the Cook PVI of D+4.1 and the 3W-RDM of D+3.2, and the projected Nevada vote based on the 538 PL was D+5.5.

**********

In this post, I assessed the validity of one of my baseball player performance metrics—the Index of Offensive Ability—by comparing it to two other commonly-used statistics, OPS+ and WAR. Here is how I described validity in that post:

Validity is the extent to which an index/measure/score actually measures what it is designed to measure, or “underlying construct”. While now considered a unitary concept, historically, there were three broad approaches to “assessing” validity: content, construct and criterion.

 Content validity is the extent to which an index/measure/score includes the appropriate set of components (not too many, not too few) to capture the underlying construct (say, a state’s partisan “lean”). Construct validity is how strongly your index/measure/score relates to other indices/measures/scores of the same underlying construct, including a priori expectations of what values should be (sometimes called face validity). Criterion validity considers how well outcomes “predicted” by the index/measure/score align with the actual outcomes.

As you have probably guessed by now, I will spend the rest of this post comparing my 3W-RDM to the Cook PVI and the 538 PL.

But first, I offer a mea culpa.

Before my “Democratic blue wall thesis” post in February 2017, I had used the 3W-RDM (which did not even have a name until then) only for my own edification and amusement. That, however, does not excuse me for not even attempting to validate this measure until now. Moreover, I should not have started writing data-driven posts using the 3W-RDM—implicitly asserting its validity without empirical evidence—until I had performed that validation.

I now present that empirical validation evidence.

Content validity: All three measures not only use presidential election voting data, but they also compare state and national margins in some way. This makes sense because presidential elections feature one party nominee advocating (theoretically) the same platform in every state. By comparison, other statewide elections (governor, Senate) feature candidates who share a party label yet may have very different policy stances. While this may be less true now for Senate races, which are becoming more nationalized, there is still a vast difference between Democratic Senators like Joe Manchin of West Virginia and Elizabeth Warren of Massachusetts, and between Republican governors like Charlie Baker of Massachusetts and Sam Brownback of Kansas.

Thus, despite differences in number of elections utilized, weighting and margin calculation, all three measures arguably have high content validity.

Construct validity. A correlation coefficient (“r”) is a number between -1.00 and +1.00 indicating how two variables co-relate to each other in a linear way[5]. If every time one variable increases, the other variable increases, that would be r= +1.00, and if every time one variable increases, the other variable decreases, that would be r=-1.00. R=0.00 means there is no linear association between the two variables.

I calculated the projected presidential election margin (D% total vote – R% total vote) in each state (plus DC) in every presidential election from 1996 through 2016 by adding each state’s partisan lean score before that election to the actual national popular vote margin. In other words, I repeated the example of Nevada (projected 2016 presidential vote: Cook PVI=D+6.2, 538 PL=D+5.5, 3W-RDM=D+5.2) for all 306 state-level presidential election margins.

Here are the average correlations (PVI vs. PL, PVI vs. 3W-RDM, PL vs. 3W-RDM) between the three sets of projected margins in each election year:

1996    +0.995

2000    +0.994

2004    +0.997

2008    +0.998

2012    +0.997

2016    +0.999

Clearly, each partisan lean measure is nearly identically capturing the underlying partisan distribution of states from most to least Democratic, indicating that each measure has very high construct validity.

Criterion validity. Building upon the analysis of construct validity, the simplest way to assess criterion validity is to compare the projected presidential election margin in each state in each year to the actual margins.

Table 1 does this for each state in 2016. A negative difference means the state voted less Democratic than expected, and a positive difference means the state voted more Democratic than expected. States are sorted from most “less Democratic” to most “more Democratic.”

Table 1: Differences Between Projected and Actual State-Level Presidential Vote Margin (Democratic % – Republican %), 2016

State Cook PVI 538 PL 3W-RDM Mean
West Virginia -17.8% -15.8% -20.0% -17.8%
North Dakota -17.6% -16.2% -16.6% -16.8%
Iowa -13.7% -13.5% -13.5% -13.6%
South Dakota -12.7% -11.6% -12.5% -12.3%
Maine -10.2% -10.2% -10.1% -10.2%
Missouri -10.1% -8.8% -10.7% -9.9%
Indiana -10.8% -9.0% -9.0% -9.6%
Michigan -9.8% -8.8% -9.1% -9.3%
Rhode Island -9.1% -9.4% -9.1% -9.2%
Ohio -8.4% -8.8% -8.9% -8.7%
Montana -8.4% -6.8% -7.4% -7.5%
Wisconsin -7.8% -6.8% -7.1% -7.2%
Hawaii -9.0% -8.6% -3.9% -7.1%
Kentucky -6.5% -6.1% -7.9% -6.9%
Vermont -7.2% -6.9% -5.2% -6.4%
Delaware -7.2% -6.3% -5.8% -6.4%
Wyoming -5.0% -5.0% -6.7% -5.6%
Tennessee -4.5% -4.3% -6.6% -5.1%
Pennsylvania -5.1% -4.7% -5.4% -5.1%
Minnesota -4.1% -4.2% -4.5% -4.3%
New Hampshire -3.8% -3.6% -4.0% -3.8%
Nevada -3.8% -3.1% -2.8% -3.3%
Alabama -2.0% -3.1% -3.3% -2.8%
Mississippi -1.8% -3.3% -2.6% -2.5%
Connecticut -2.9% -2.3% -2.4% -2.5%
Arkansas -1.0% -1.6% -5.0% -2.5%
Nebraska -2.8% -2.4% -1.8% -2.3%
New York -1.8% -2.7% -1.8% -2.1%
South Carolina -0.9% -1.6% -1.4% -1.3%
Oklahoma -0.4% -0.8% -2.1% -1.1%
New Mexico -1.2% -0.6% 0.1% -0.5%
Illinois -0.9% 0.6% 0.2% 0.0%
Florida 0.5% 0.1% 0.1% 0.2%
New Jersey 0.7% -0.6% 0.7% 0.3%
Oregon -0.1% 0.4% 0.6% 0.3%
Louisiana 2.1% 0.5% -0.6% 0.7%
North Carolina 0.8% 0.4% 1.2% 0.8%
Idaho 1.2% 0.9% 0.7% 1.0%
Colorado 1.2% 1.3% 1.9% 1.4%
Kansas 1.8% 2.1% 1.4% 1.8%
Washington 2.9% 3.0% 3.3% 3.1%
Maryland 3.7% 3.1% 4.6% 3.8%
Virginia 3.7% 3.5% 4.5% 3.9%
DC 4.3% 5.2% 4.8% 4.8%
Georgia 5.1% 4.7% 5.1% 5.0%
Massachusetts 5.8% 6.0% 4.7% 5.5%
Alaska 7.2% 3.8% 5.6% 5.5%
Arizona 9.0% 8.0% 7.4% 8.1%
Texas 8.5% 8.4% 8.5% 8.5%
California 9.4% 9.3% 10.6% 9.8%
Utah 24.8% 27.6% 24.8% 25.7%
Mean -2.3% -2.1% -2.3% -2.2%

On average, the measures overestimated Clinton’s performance by a relatively low 2.2 percentage points, with no meaningful difference across measures. Five states—West Virginia, North Dakota, Iowa, South Dakota and Maine—were at least 10 percentage points less Democratic than projected using all three measures; Clinton still won Maine, but by “only” 3.0 percentage points. Four states—Utah, California, Texas and Arizona—were at least seven percentage points more Democratic than projected using all three measures; Clinton won only California of this group, though there are signs that Texas and, especially, Arizona are becoming more Democratic. The massive disparity in Utah  results from the presence of unaffiliated presidential candidate Evan McMullin, a Utah native, on the ballot; his 21.3% of the vote cut deeply into Trump’s vote, so the latter “only” won the state by 17.9 percentage points.

 As Table 2 shows, the performance of these measures—using the average of the actual difference in margins—was the worst since 2000, when they also overestimated Democratic performance by an average of 2.2 percentage points. On average, across all six presidential elections, these measures overestimated Democratic performance by just 0.9 percentage points, a solid performance.

Table 2: Average Difference Between Projected and Actual State-Level Presidential Vote Margin (Democratic % – Republican %), 1996-2016

Year Cook PVI 538 PL 3W-RDM Mean
1996 -0.7% -1.0% -0.9% -0.9%
2000 -2.0% -2.2% -2.5% -2.2%
2004 0.1% 0.4% -0.3% 0.1%
2008 0.7% 0.5% 0.4% 0.5%
2012 -0.7% -0.8% -0.6% -0.7%
2016 -2.3% -2.1% -2.3% -2.2%
Mean -0.8% -0.9% -1.0% -0.9%

These values can be deceptive, however. Consider the performance of the 3W-RDM in 2016. It overestimated Clinton’s margin in Montana by 7.4 percentage points, and it underestimated her margin in Arizona by an identical 7.4 percentage points. In both states the difference was 7.4 percentage points, but averaging the two (0.0 percentage points) would suggest that the 3W-RDM was spot on.

In fact, the three measures missed the actual presidential election margin by at least five percentage points in 26 states.

Table 3 resolves this problem by displaying the average absolute value of the difference between the projected and actual presidential election margins.

Table 3: Average of Absolute Value of Differences Between Projected and Actual State-Level Presidential Vote Margin (Democratic % – Republican %), 1996-2016

Year Cook PVI 538 PL 3W-RDM Mean
1996 5.4% 5.1% 5.6% 5.4%
2000 5.5% 5.9% 6.8% 6.1%
2004 3.9% 3.6% 4.2% 3.9%
2008 6.3% 5.7% 6.2% 6.1%
2012 3.3% 3.2% 3.5% 3.3%
2016 5.9% 5.6% 5.9% 5.8%
Mean 5.0% 4.8% 5.4% 5.1%

On average, the projected and actual presidential election margins differed by 5.1 percentage points in either direction. The 3W-RDM, which differed by an average of 5.4 percentage points, fared slightly worse than the Cook PVI and 538 PL. The best years for these measures were two re-election years, 2004 (3.9 percentage points) and 2012 (3.3), and the worst years were the open seat elections of 2000, 2008 (both 6.1) and 2016 (5.8). The overall worst performance was the 3W-RDM in 2000 (6.8), while the overall best performance was the 538 PL in 2012 (3.2).

I performed identical analyses to those summarized in Tables 2 and 3 using two alternate versions of the 3W-RDM, one which used a 1-3-5 weighting scheme and one which weighted all three years equally. The results were nearly identical to those shown here (though the non-weighted 3W-RDM tended to perform worse on the absolute value differences), suggesting that if the 3W-RDM is slightly less “predictive” than the other two measures, it is not due to the weighting scheme but (most likely) to the inclusion of data from a third election year.

Finally, I counted how many—and which—states were “called” incorrectly by each measure in each presidential election.

Table 4: “Mis-called” States, 1996-2016

Year Cook PVI* 538 PL 3W-RDM Average
1996 9

AZ, CO, FL, MT, NV, NH, NC, SD, TX

8

AZ, CO, FL, GA, MT, NC, SD, TX

9

AZ, CO, FL, GA MT, NH, NC, SD, TX

8.7
2000 5

AR, CT, LA, MO, WV

6

AR, CT, LA, MO, NH, WV

5

AR, CT, LA, MO, WV

5.3
2004 4

NH, OR, PA, WI

3

NH, OR, WI

3

NH, OR, WI

3.3
2008 4

AR, IN, MO, NC

4

AR, IN, MO, NC

7

AR,AZ, IN, MO, NC, VA, WV

5.0
2012 0 1

FL

0 0.3
2016 5

IA, MI, OH, PA, WI

5

IA, MI, OH, PA, WI

5

IA, MI, OH, PA, WI

5.0
Mean 4.3 4.5 4.8 4.5

        *States in boldface were “predicted” Democratic wins, and states in italics were

         “predicted” Republican wins.

On average, four or five (out of 51) states are “mis-called” in a given presidential election. Again, the 3W-RDM fared slightly worse (4.8) than average (4.5). Of the 83 total misses (out of 918 possibilities), 52 (62.7%) were states that were projected Democratic wins that were actually won by the Republican nominee.

The presidential election of 1996, when Democrat Bill Clinton cruised to an easy reelection, had the most mis-called states, eight or nine; seven states (Arizona, Colorado, Florida, Montana, North Carolina, South Dakota, Texas) were mis-called by all three measures. By contrast, only one state was mis-called in 2012, Florida by the 538 PL: it projected Obama would lose Florida by 0.1 percentage points when he in fact won it by 0.9 percentage points.

Despite these differences, I would argue that all three measures have high criterion validity, as each does a reasonably good job of “projecting” the actual presidential election margin in a given state and year. My 3W-RDM performed only slightly worse than the other two measures, so I will stick with it for now.

**********

One final note about the utility of partisan lean measures.

The Alabama special Senate election between Republican Roy Moore and Democrat Doug Jones to be held on December 12, 2017 is drawing national attention for two reasons. One, a win by Jones would reduce the Republican Senate majority to 51-49. Two, Moore has been dogged by allegations of sexual misconduct with minors (as well as having been removed twice as Alabama’s Chief Justice for defying federal court orders).

The public polls of this election, which once showed a Moore lead of ~11 percentage points, have tightened considerably since the allegations first appeared on November 9, 2017. As of now, depending on how you aggregate and weight these polls, Moore is somewhere between four percentage points ahead and one percentage points ahead; my best estimate is that Moore is ahead 1.7 percentage points.

But consider this. Following the 2016 presidential election, the average partisan lean for Alabama (using all three measures) is D-28.7. As of this writing, the best estimate of how Democrats will fare in the 2018 Congressional elections is that they are ahead by 7.8 percentage points.

Putting these two values together implies that a generic Republican Senate candidate should be leading a generic Democratic Senate candidate by 20.9 percentage points (28.7 minus 7.8): this should not even be a close contest.

However, the polls suggest that Jones is performing somewhere between 16.9 and 21.9 percentage points better than a generic Democrat—that is a stunning difference, and one that may bode very well for Democrats in 2018.

Until next time…

[1] Technically, he only won 111, as one Democratic elector in Washington (state) cast his presidential vote for Lloyd Bentsen, the 1988 Democratic nominee for vice president, and cast his vice presidential vote for Dukakis.

[2] I freely confess to being the artist. This kid-friendly (fine, I had just turned 22) exhortation to vote must have been in the Comics section of the Washington Post (I was living in DC at the time) the Sunday before the 1988 elections.

[3] My data start in 1984, so I would only have 5W-RDM for 1984-2000, 1988-2004, 1992-2008, 1996-2012 and 2000-2016.

[4] I have experimented with adding a weighted linear trend to the 3W-RDM. The logic is that if I want to use the previous three election margins in a state to “forecast” the state margin in the next election, I should account for the fact that, over time, some states are growing relatively more Democratic (e.g., Nevada has become 11.7 percentage points more Democratic relative to the nation since 1984-1992) or less Democratic (e.g., West Virginia, 44.7 percentage points). Adding a weighted average of all previous election-to-election changes in RDM to a 3W-RDM would, theoretically, account for any increased partisanship over the ensuing four years. For the analyses below, however, there was very little difference between the 3W-RDM and the 3W-RDM+weighted linear trend, so I exclude it.

[5] More formally r = covariance(x,y) divided by SD(x) * SD(y).

The 2016 U.S. presidential election viewed through one statistic

The 2016 United States (U.S.) presidential election is one of those elections (1948, 1960, 1968 and 2000 also come to mind) people will be re-hashing as long as the U.S. continues to HAVE presidential elections. I have already shared data-driven thoughts on the 2016 U.S. presidential election here, here, here, here, here and here.

Grounding my thoughts about this election is the following sequence of data points (drawn from Dave Leip’s invaluable Atlas of U.S. Presidential Elections):

  • Democratic presidential nominee Hillary Clinton won 2,868,518 more votes OVERALL than Republican presidential nominee Donald Trump (48.0% vs. 45.9%).
  • Trump won the election because he won more Electoral College votes (EV; 306 to 232[1])
  • Trump won more EV because he won narrow victories in three states:
    • Pennsylvania (20 EV): 44,292 votes, or 0.72%
    • Wisconsin (10 EV): 22,748, or 0.76%
    • Michigan (16 EV): 10,794, or 0.22%
  • Trump won because of just 77,744 votes in three closely-fought states, or 0.057% of the 137,125,484 votes cast in the 2016 U.S. presidential election.

I want to shout these numbers whenever political pundits or elected officials and their allies fret about “how Democrats can ever win back voters in 2018 or 2020.”

To all those folks I say, Chill! The 2016 U.S. presidential election was VERY close, not to mention that Democrats also netted two U.S. Senate seats and six U.S. House of Representatives seats that year.

And while it is absolutely true that, relative to the extraordinarily Democratic years of 2006 and 2008, Democrats have been losing ground badly at the state level (with 2017 election results suggesting a slow-moving reversal), that is not the focus of this post.

Instead, I want to focus on the single statistic that strikes me as the key to understanding the outcome of the 2016 U.S. presidential election.

**********

First, however, just bear with me while I briefly address “electoral legitimacy” arguments made about that election.

These basically fall into two groups:

  1. Russian cyberattacks amplified through American social and traditional media
  2. Voter suppression efforts

The goal of the Russian cyberattacks (including, but not limited to, hacking Democratic National Committee e-mails and releasing them through WikiLeaks; purchasing thousands of ads on social media platforms; coordinating “trolling” on those same social media platforms by Russian nationals) appears to have been to sow discord in the American electorate; punish 2016 Democratic presidential nominee Hillary Clinton; and, PERHAPS, promote the candidacy of Republican presidential nominee Donald Trump (with or without “collusion” on their part).

That such meddling did occur is widely accepted, even if the efficacy of that meddling is debatable.

But the next question to be asked is this: as a result of this interference, how many voters who would otherwise have voted for Clinton did not vote for her, regardless of whether they voted for somebody else or simply did not cast a presidential vote at all?

This counterfactual may not be possible to assess given the voting data at our disposal and the multitude of reasons we choose one candidate over another.

Well, besides simple partisanship that is (data source found by clicking on election year):

Table 1: Percentage of Self-Identified Partisans Who Voted For Presidential Candidate of Their Party, 2000-2016

Election % Democrats

voting Democratic

% Republicans voting Republican Margin among Independents
2016 89%

(36% of electorate)

88%

(33%)

+4% Republican

(31%)

2012 92%

(38%)

93%

(32%)

+5% Republican

(29%)

2008 89%

(39%)

90%

(32%)

+8% Democratic

(29%)

2004 89%

(37%)

93%

(37%)

+1% Democratic

(26%)

2000 87%

(39%)

91%

(35%)

+2% Republican

(26%)

Mean 89%

(38%)

91%

(34%)

Margin=4%

(28%)

In the previous five presidential elections, 87-92% of self-identified Democrats voted for the Democratic nominee, and 88-93% of self-identified Republicans voted for the Republican nominee. Self-identified Independents (whose share of the electorate seems to be increasing over time), most of whom usually cast their ballots for the same party over time, divided their votes fairly evenly between the Democratic and Republican nominees (while also being more likely to choose a third-party option[2]) over these same elections.

American politics is highly polarized, and the vast majority of voters simply vote for the nominee with the same party identification as them, so the pool of voters who would have been swayed by Russian interference was already very small.

Again, that is not to say the meddling did not occur, that it was not an attack on our sovereign democracy, and that no votes were changed from “Clinton” to either “not Clinton” or a non-vote. I just think there is a far less “conspiratorial” way to understand the results of the 2016 U.S. presidential election.

As for voter suppression efforts like restrictive voter ID laws, fewer polling places and shorter/no early voting periods, there is some evidence that this occurred in states highly relevant to the outcome of the 2016 U.S. presidential election, including Wisconsin and North Carolina.

Yes, I wrote “North Carolina.”

While the “path of least resistance” for Clinton would have been to flip just under 78,000 votes in three “Rust Belt” states, an alternate path would have been to flip just 285,826 votes (0.21%) in two southeastern states: Florida (Clinton -112,911, or 1.2%) and North Carolina (Clinton –172,915, or 3.6%). Or to flip 157,203 votes (0.11%) in Florida and Pennsylvania…you get the idea.

But, even IF Wisconsin and North Carolina had voted for Clinton if voter suppression had not existed (a difficult counterfactual to prove), that would only have garnered Clinton 25 additional EV, increasing her total to 257, 13 shy of the 270 required for victory. She would still have needed to win one of Michigan, Pennsylvania or Florida, states where there have been no claims of voter suppression of which I am aware.

The point is, while Russian interference and voter suppression certainly happened, demonstrating that they prevented enough votes for Clinton in the right combination of states to deny her an Electoral College victory in the 2016 U.S. presidential election is extremely difficult. The simple fact that each was attempted is pernicious enough.

**********

What makes the 2016 U.S. presidential election stand out from the pack is how disliked both major party nominees were.

totalfavunfavehorizontalAccording to the exit polls, Clinton was viewed favorably by 43%, and unfavorably by 55%, of the 2016 presidential electorate; the corresponding values for Trump were 38% and 60%, respectively. These line up nicely with the RealClearPolitics (RCP) averages going into Election Day (November 8, 2016): Clinton 42%/54%, Trump 38%/58%.[3].

On average, 95% of those with a favorable view of a candidate voted FOR that candidate. Among voters with an unfavorable view of Clinton, 81% voted for Trump, and among voters with an unfavorable view of Trump, 77% voted for Clinton.

Here is the kicker, however:

An unusually high 18%[4] of the electorate had an unfavorable view of both Clinton AND Trump. This pivotal portion of the electorate gave 47% of their votes to Trump, 30% to Clinton and 23% to neither candidate.

That’s right, Trump won by 17% percentage points nationwide among voters who disliked BOTH major-party candidates.

And the support for Trump among this portion of the electorate was much stronger in the six states Clinton lost by less than four percentage points (total EV=99):

Table 2: Favorability Ratings for Clinton and Trump in Six Key States, 2016

State EV Trump Margin Clinton Trump Both Unfavorable Margin among

Both Unfavorable

MI 16 +0.2% 42/56 39/59 20% Trump +21%
PA 20 +0.7% 42/57 42/56 17% Trump +25%
WI 10 +0.8% 42/56 35/64 22% Trump +37%
FL 27 +1.2% 45/53 41/57 14% Trump +37%
AZ 11 +3.5% 41/57 41/57 18% Trump +17%
NC 15 +3.6% 43/56 41/58 16% Trump +36%
Mean 16 +1.7% 43/56 40/59 18% Trump +29%

On average, 18% of the voters in these six states had an unfavorable view of both Clinton and Trump, with Clinton earning 27% of their votes (3 percentage points lower than nationwide) and Trump earning 56% of their votes (9 percentage points higher than nationwide). Third-party candidates did worse (18%), on average, than nationwide (23%) with this group in these six states; the exception is Arizona (29%), neighbor to the west of 2016 Libertarian presidential nominee Gary Johnson’s home state of New Mexico[5].

In fact, Trump received an astonishing 60% of the “pox on both your houses” votes in Wisconsin, 61% in Florida and 62% in North Carolina.

I can find no historical data to which to compare these numbers, so I do not know what a typical vote distribution among this segment of the electorate is. Still, it is important to keep in mind that the 2016 U.S. presidential election took place after eight years with one party (Democrats) occupying the White House and no incumbent running. Voters often look to change White House control in these elections: prior to 2016, of the six such elections starting with 1960, the party not occupying the White House had won five of them (1960, 1968, 1976, 2000, 2008). The exception was 1988, when Republican nominee George H. W. Bush beat Democratic nominee Michael Dukakis by 7.7 percentage points and 315 EV.

These elections also tend to be very close, with the party not occupying the White House winning the two-party vote by an average of just 0.3 percentage points and 22 EV (excluding 1988, these values are 1.9 percentage points and 90 EV)[6].

According to the RCP average, voters on Election Day 2016 felt the country was going in the wrong direction by a margin of 61-31%. Combine this with an eight-year/no-incumbent election and Clinton (or any Democratic presidential nominee) should always have been seen as a slight underdog. The historic unpopularity of Trump (net -21 percentage points) may have led observers to conclude that this election would be different, but they did not take into account Clinton’s only-marginally-better favorability rating (-13 percentage points).

Still, it is worth considering two alternate scenarios in the six states listed in Table 2:

  1. The voters disliking both Clinton and Trump give the same support to “other” candidates, but split the two-party votes EVENLY between Clinton and Trump.
  2. The distribution of the “pox on both houses” vote in these six states matches the nation (30% Clinton, 47% Trump, 23% Other)

Table 3 lists how each state would have voted under both scenarios, with the state winner in bold italics.

Table 3: Statewide Vote Distributions in Six States, 2016, Under Three Methods of Splitting Votes of Clinton-Trump Disapprovers

State Actual 2016 results 2-party vote split even Votes split 30-47-23
Clinton Trump Clinton Trump Clinton Trump
MI 47.0% 47.2% 49.1% 45.1% 47.2% 46.6%
PA 47.5% 48.2% 49.6% 46.0% 47.3% 46.6%
WI 46.5% 47.2% 50.5% 43.1% 48.0% 44.4%
FL 47.4% 48.6% 50.0% 46.0% 48.2% 46.6%
AZ 44.6% 48.1% 46.1% 46.5% 45.1% 48.6%
NC 46.2% 49.8% 49.1% 46.9% 46.8% 47.4%

Under both scenarios, Clinton would not only have won Michigan, Pennsylvania and Wisconsin (giving her 278 EV, 8 more than necessary), she also would have won Florida’s 27 EV, for a total of 305 EV. North Carolina’s 15 EV would also have gone to Clinton if the voters who disapproved of her and Trump had split their two-party votes evenly. Arizona, because of its relatively high 7.3% of the vote for neither Clinton nor Trump, would still have gone to Trump under both scenarios.

In other words, the 2016 U.S. presidential was an eight-year/non-incumbent election featuring two historically unpopular candidates. Neither major party candidate had a net positive favorable rating, resulting in an unusually high 18% of the electorate disliking both. Given that this was a change election (net -31% felt country on wrong track), it is not surprising in retrospect that this key bloc of voters chose the Republican nominee (the nominee of the party not occupying the White House), propelling him to the White House.

Still, had the Democratic presidential nominee been viewed even a little more favorably, she might easily have won four additional states with a combined 73 EV, thus winning the White House.

And here is where, if one were to squint hard enough, one could construct an argument that looked something like this:

There is evidence from the RCP averages that Clinton’s net favorability—which was roughly even in June 2015, just as the 2016 U.S. presidential election was beginning—steadily worsened after that, landing at 13 percentage points unfavorable by November 2016. Trump’s net unfavorability, meanwhile, hardly changed over this same period. This could be seen as evidence that Russian interference had the effect of slowly increasing her net unfavorability, to the point where voters nearly disapproved equally of both candidates (then opted for the nominee of the party not occupying the White House).

While this is…plausible, there is one profound flaw (other than the simple fact of NOT explaining why voters who disapproved of both Clinton and Trump then voted heavily for Trump). On January 23, 2013, Clinton was viewed favorably by 63% of American voters and unfavorably by 28%, for a net favorability of 35 percentage points. She had just stepped down from her perch as a popular Secretary of State and was publicly undecided about her future in electoral politics. Still, from that day forward, her net favorability declined steadily and inexorably to nearly even in the spring and summer of 2015.

That is, Clinton was becoming more unpopular long before ANY Russian interference in the 2016 U.S. presidential election. Moreover, her net unfavorability actually hit its nadir (18% net unfavorable) in late May 2016. After that, while the percentage disapproving of Clinton changed little, the percentage approving of her steadily increased.

To me, the bottom line is this:

Democrats are best served understanding that 2016 was a change election featuring two historically unpopular major party nominees. Under those circumstances, an unpopular nominee of the party not occupying the White House is almost certain to beat an unpopular nominee of the party occupying the White House. Period.

Focusing on Russian interference and/or voter suppression as the “causes” of Clinton’s defeat is a wild goose chase. Both are antithetical to a well-functioning, mature democracy and need to be investigated and prevented to the maximum extent, but they also distract from the fact that 46% of the American electorate were predisposed to accept Trump’s message.

Democrats should also realize that Clinton actually defied recent presidential election history by winning the popular vote by just over two percentage points, and that there are strong reasons for optimism in 2018 and 2020 given their growing strength with white college-educated voters, especially women.

In other words, Chill!

Until next time…

[1] Technically, 304-227, as seven Electors voted for other candidates.

[2] In 2016, for example, 12% of self-identified Independents voted for a non-major-party candidate, as opposed to just  3% of self-identified Democrats and 4% of self-identified Republicans.

[3] Given that nearly every poll included in the final RCP averages was of “likely voters,” pollsters did a very good job modeling the actual electorate. This is also indirect evidence that voter suppression did not, in fact, keep an electorally-significant number of Democratic voters from the polls: the projected electorate looked like the actual electorate.

[4] I base this assertion on 1) the fact that voting preferences of voters with an unfavorable view of both major-party candidates had not been assessed prior to 2016 and 2) the historic unpopularity of Clinton and Trump.

[5] Excluding Arizona yields a Clinton 27%, Trump 58%, Other 16% split among the 18% of voters disliking both Clinton and Trump

[6] The fact that Clinton won the popular vote by 2.1 percentage points is even more remarkable in this context, while her 77 EV vote loss was about in line with expectations (22-90 EV loss).

Positively pondering pesky probabilities, perchance

One inspiration to start this “data-driven storytelling” blog was the pioneering work of Nate Silver and his fellow data journalists at FiveThirtyEight.com; their analyses are an essential “critical thinking” reality check to my own conclusions and perceptions. Indeed, when I finally get around to designing and teaching my course on critical thinking (along with my film noir course), the required reading would include Silver’s The Signal and the Noise and a deep dive into Robert Todd Carroll’s The Skeptic’s Dictionary. I will also include Ken Rothman’s Epidemiology: An Introduction; what drew me to epidemiology (besides my long career as a public health data analyst) was its epistemological aspect. By that I mean how the fundamental methods and principals of epidemiology allow us to critically assess any narrative or story.

To that end, I have been reading with great interest Silver’s 11-part series that “reviews news coverage of the 2016 general election, explores how Donald Trump won and why his chances were underrated by most of the American media.” And while I highly recommend the entire series of articles, the September 21 conclusion is the jumping off point for my own observations about assessing the likelihood of various events.

**********

Let me begin with a passage from that article:

In recent elections, the media has often overestimated the precision of polling, cherry-picked data and portrayed elections as sure things when that conclusion very much wasn’t supported by polls or other empirical evidence.

I personally think investigative journalists are heroic figures who will ultimately save American democracy from its current self-induced peril. But they are trained in a very specific way: deliver the fact of a story with certainty and immediacy. In so doing, they are responding to media consumers with little patience for complex narratives suffused with uncertainty.

To quote Silver again, “a story can be 1. fast, 2. interesting and/or 3. true — two out of the three — but it’s hard for it to be all three at the same time.”

One narrative that developed fairly early about the 2016 presidential election campaign was that Democratic nominee Hillary Clinton was the all-but-inevitable victor. I wrote about one version of this flawed narrative here.

Reinforcing this narrative were election forecasts issued during the last weeks of the campaign that practically said “stick a fork in Trump, he is finished.” But as Silver rightly observes, some of these models were flawed because they failed to account for the “correlation in outcomes between [demographically similar] states.” For example, were Republican nominee Donald Trump to outperform his polls in Wisconsin on Election Day, he would likely also do so in Michigan, Minnesota and Iowa. And that is essentially what happened.

Still, because aggregating polls yields a more precise picture of the state of an election at a given point in time, I aggregated these 2016 election forecasts. Going into Election Day, here were some estimated probabilities of a Clinton victory, ranked lowest to highest.

FiveThirtyEight 71.4%
Betting markets 82.9%[1]
The New York Times Upshot 84.0%
DailyKos 92.0%
HuffingtonPost Pollster 98.2%
Princeton Election Consortium (Sam Wang) 99.5%

The average and median forecast was 88.0%. Remove the most skeptical forecast (though Clinton still a 5:2 favorite), and the average and median jump to 91.3% and 92.0%, respectively. By contrast, if you remove the least forecast, the average and median drop to 84.1% and 83.5%, respectively.

It is an understandable human tendency to look at a probability over 80% and “round up” from “very likely, but not guaranteed” to “event will happen.” And, under the frequentist definition of probability, we would be correct more than 80% of the time in the long run.

But we would not be correct as much as 20% of the time.

Ignoring Wang’s insanely optimistic forecast for various reasons, the “aggregate” forecast I had in mind on Election Day was that Clinton had about an 84% chance of winning.

The flip side, of course, was that Trump had about a 16% chance of winning.

A good way to interpret this probability is to think about rolling a fair, six-sided die.

Pick a number from one to six. The chance that if you roll the die, the number you picked will come up, is 1 in 6, or 16.7%.

On Election Day, Trump metaphorically needed to roll his chosen number…and he did.

But even if take the Wang-inclusive average of 88%, that is still a 1 in 8 chance. Throw eight slips of paper with the numbers one through eight written on them in a hat (I like fedoras, myself), pick one and draw. If your number comes up (which will happen 12% of the time over many draws), you win.

Trump picked a number between one and eight then pulled it out of our hypothetical fedora, and he won the election.

One way people misunderstand probability (and one of many reasons I am resolutely opposed to classical statistical significance testing) is mentally converting event x has a very low probability (like, say, matching DNA in a murder trial—only a 1 in 2 million chance!) with that event cannot happen.

So, even the Wang forecast—which gave Trump only a 1 in 200 chance of winning—did NOT mean that Clinton would definitely win. It only meant that Trump had to pull a specific number between one and 200 out of our hypothetical fedora. He did, and he won.

**********

On the other end of the spectrum is an overabundance of caution in assessing the likelihood of an event. This usually occurs when interpreting election polls.

In this post, I discussed Democratic prospects in the 2017 and 2018 races for governor.

One of the two governor’s races in November 2017 is in Virginia, where Democratic governor Terry McAuliffe is term-limited. The Democratic nominee is Lieutenant Governor Ralph Northam, and the Republican nominee is former Republican National Committee chair Ed Gillespie.

Here are the 13 public polls of this race listed on RealClearPolitics.com[2] taken after the June 13, 2017 primary elections:

Poll Date Sample MoE Northam (D) Gillespie (R) Spread
Monmouth* 9/21 – 9/25 499 LV 4.4 49 44 Northam +5
Roanoke College* 9/16 – 9/23 596 LV 4 47 43 Northam +4
Christopher Newport Univ.* 9/12 – 9/22 776 LV 3.7 47 41 Northam +6
FOX News* 9/16 – 9/17 507 RV 4 42 38 Northam +4
Quinnipiac* 9/14 – 9/18 850 LV 4.2 51 41 Northam +10
Suffolk* 9/13 – 9/17 500 LV 4.4 42 42 Tie
Mason-Dixon* 9/10 – 9/15 625 LV 4 44 43 Northam +1
Univ. of Mary Washington* 9/5 – 9/12 562 LV 5.2 44 39 Northam +5
Roanoke College* 8/12 – 8/19 599 LV 4 43 36 Northam +7
Quinnipiac* 8/3 – 8/8 1082 RV 3.8 44 38 Northam +6
VCU* 7/17 – 7/25 538 LV 5 42 37 Northam +5
Monmouth* 7/20 – 7/23 502 LV 4.3 44 44 Tie
Quinnipiac 6/15 – 6/20 1145 RV 3.8 47 39 Northam +8

Eight of these polls have Northam up between four and seven percentage points, including four of the last six. Two polls show a tied race. No poll gives Gillespie the lead.

And yet, here was the headline on Taegan Goddard’s otherwise-reliable Political Wire on September 19, 2017, referring to the just-released University of Mary Washington (Northam +5) and Suffolk polls (Even): Race For Virginia Governor May Be Close.

Granted, the two polls gave Northam an average lead of only 2.5 percentage points, which, without context, suggest a close race on Election Day. Furthermore, all three Political Wire Virginia governor’s race poll headlines since then have been on the order of: Northam Maintains Lead In Virginia.

Here is the thing, however. Most people (as I did) will equate “close” with “toss-up.” But there is a huge difference between “we have no idea who is going to win because the polls average out to a point or two either way” and “one candidate consistently has the lead, but the margin is relatively narrow.”

The latter is clearly the case in the 2017 Virginia governor’s race, with Northam’s lead averaging 4.4 percentage points in eight September polls within a narrow range (standard deviation [SD]=3.3). We are still more than five weeks from 2017 Election Day (November 7), so this is unlikely to be “herding,” the tendency of some pollsters to adjust their demographic weights and turnout estimates to avoid an “outlier” result (undermining the rationale for aggregating polls in the first place).

The problem comes when members of the media try to interpret the results of individual polls. They have absorbed the lesson of the “margin of error” (MoE) almost too well.

For example, the Monmouth poll conducted September 21-25, 2017 gives Northam a five percentage point lead, with a 4.4 percentage point MoE. Applying that MoE to both candidates’ vote estimates, we have 95% confidence that the “actual” result (if we had accurately surveyed every likely voter, not a sample of 499) is somewhere between Gillespie 48.4, Northam 44.6 (Northam down 3.8) and Northam 53.4, Gillespie 39.6 (Northam up 13.8). It is this range of possible outcomes, from a somewhat narrow Gillespie victory to a comfortable Northam win that leads members of the media to imply through oversimplification that this race will be close, meaning “toss-up.”

And yet, even within this poll, the probability (using a normal distribution, mean= 5.0, SD=4.4) that Northam is as little as 0.0001 percentage points ahead is 87.2%, making him a 7:1 favorite, about what Hillary Clinton was on Election Day 2016.

OK, maybe that was not the best example…

But when you aggregate the eight September polls, the MoE drops to about 1.3[3], putting the probability Northam is ahead at well over 99%. Even if the MoE only dropped to 3.0, the probability of a Northam lead would still be about 93%.

My point is this. Every poll needs to be considered not just as an item in itself (polls as NEWS!) but within the larger context of other polls of the same race. And in the 2017 Virginia governor’s race, the available polling paints a picture of a narrow but durable lead for Northam.

I have no idea who will be the next governor of Virginia. But a careful reading of the data suggests that, as of September 29, 2017, Lt. Governor Ralph Northam is a heavy favorite to be the next governor of Virginia, despite being ahead “only” 4 or 5 percentage points.

**********

Finally, here is an update on this post about the Democrats’ chances of regaining control of the United States House of Representatives (House) in 2018.

Out of curiosity, I built two simple linear regression models. One estimates the number of House seats Democrats will gain in 2018 only as a function of the change from 2016 in the Democratic share of the total vote cast in House elections. The Democrats lost the total 2016 House vote by 1.1 percentage points, so if they were to win the 2018 House vote by 7.0 percentage points, that would be an 8.1 percentage point shift.

Right now, FiveThirtyEight estimates Democrats have an 8.0 percentage point advantage on the “generic ballot” question (whether a respondent would vote for the Democratic or the Republican House candidate in their district if the election were held today).

My simple model estimates a pro-Democratic House vote shift of 9.1 percentage points would result in a net pickup of 26.7 House seats, a few more than the 24 they need to regain control. The 95% confidence interval (CI) is a gain of 17.0 to 36.4 seats.

But the probability that Democrats net AT LEAST 24 House seats is 71.1%, making the Democrats 5:2 favorites to regain control of the House in 2018.

My more complex model adds a variable that is simply 1 for a midterm election and 0 otherwise, as well as the product of this “dummy” variable and the change in Democratic House vote share. I hypothesized (correctly) that this relationship would be stronger in midterm elections.

This model estimates that a 9.1 percentage point increase from 2016 in the Democratic share of the House vote would result in a net gain of 31.8 seats. However, with two additional independent variables (and only 24 data points), the 95% CI is much wider, from a loss of 7.0 seats to a history-making gain of 68.3 seats.

Still, this translates to a 66.1% probability (2:1 favorites) the Democrats regain the House in 2018.

Figure 1 shows the estimated probability the Democrats regain the House in 2018 using both models and a range of percentage point changes in House vote share from 2016.

Figure 1: Probability Democrats Control U.S. House of Representatives After 2018 Elections Based Upon the Change in Democratic Share of the House Vote, 2016-18

Democratic Probability 2018 House capture

The simple model (blue curve) gives the Democrats no chance to recapture the House in 2018 until the pro-Democratic change in vote share reaches 6.5 percentage points, after which the probability rises sharply and dramatically to a near-certainty at the 10.0 percentage point change mark. The more complex model (red curve), meanwhile, assigns steadily increasing chances for the Democrats, flipping to “more likely than not” at the 7.0 percentage point change mark; even at a truly historic 15 percentage point change, the complex model only gives the Democrats an 85.3% chance to recapture the House in 2018.

For the record, I lean toward the more complex model.

It is worth noting that in the current FiveThirtyEight estimate, 15.8% of the electorate is undecided or chose a third party candidate (when an option). If the undecided vote breaks heavily toward the party not controlling the White House in a midterm election (one way electoral “waves” form), a 66-71% would likely be an underestimate of the Democrats’ chances of regaining control of the House in 2018.

And…apropos of nothing…Happy 51st Birthday to me (September 30, 2017)!!

Until next time…

[1]  To be honest, I do not recall where I got this number from…possibly from fivethirtyeight.com or maybe from https://betting.betfair.com/politics/us-politics/…

[2] Accessed September 28, 2017

[3] The total number of voters sampled across these eight polls is 4,915, which is 9.85 times higher than the 499 sampled in the Monmouth poll. The square root of 9.85 is 3.14. Dividing 4.1 by 3.14 gives you 1.31.

Where do rank-and-file Democrats (and Independents) stand on issues right now?

In the wake of Democratic underperformance in the 2016 elections (losing the Electoral College, insufficient gains to win back the United States House of Representatives [House] or United States Senate [Senate], net loss of two governorships, hemorrhaging state legislative seats), various “autopsies” were released.

autopsies

Some autopsies reached conclusions that contradicted the finding of other autopsies (likely due to an inherent bias in the group conducting the autopsy). Left-leaning individuals (e.g., Bernie Sanders’ campaign manager Jeff Weaver) and groups (e.g., Center for American Progress) declared that the Democratic Party needed to be more responsive to its increasingly liberal and progressive base (in primaries, especially). The more centrist Third Way argued that liberals are still outnumbered by moderates and conservatives (though perhaps only in the Rust Belt states [Michigan, Ohio, Pennsylvania, Wisconsin] won by Republican presidential candidate Donald Trump). The race to be the next chairperson of the Democratic National Committee was seen as a proxy fight over this division, with Representative Keith Ellison representing the progressives and former Secretary of Labor Tom Perez representing the “establishment.” Perez won, 235-200, then immediately named Ellison deputy chairman in a nod toward party unity.

Other reports differed over whether Democrats should focus more on white men without a college degree or on younger and/or minority voters. I weighed in on this question here.  And the data journalism website FiveThirtyEight.com framed their “post-mortem” in the context of what Democrats would expect in their 2020 president.

Given this apparent divide over the best way for Democrats to proceed, which encompasses everything from messaging to election targeting to fundraising to candidate recruitment, I thought it would be a useful exercise to review what self-identified Democrats actually believe right now (along with the Independents they will need to capture to, say, win back the House in 2018). That is, what issue positions, as measured by available public polling, distinguish a majority of self-identified, rank-and-file Democrats from a majority of self-identified, rank-and-file Republicans? And, in these cases, when do a majority of self-identified, rank-and-file Independents align with the Democrats?

**********

Just bear with me while I review my methodology.

I used all polls available on the Issues page of PollingReport.com. This page breaks down non-partisan, publicly-available, issue-oriented polls into 20 categories: Problems and Priorities, Abortion, Budget and Taxes, Crime, Disaster Preparedness and Relief, Education, Energy, Environment, Food, Foreign Affairs and Defense, Guns, Health Policy, Illegal Drugs, Immigration, Law, LGBT, Race and Ethnicity, Social Security, Space Exploration, Transportation. Some categories, such as Foreign Affairs and Defense, have subcategories (e.g., Isis and Terrorism).

For each issue, I collected all polls for which partisan breakdowns (Democrat, Republican, Independent) were provided, going back (when necessary) to the summer of 2014. In this way I balance opinion recency with the desire to review as wide an array of specific issues as possible, while also capturing data from both the Trump presidency and the preceding Barack Obama presidency.

There are three important caveats about these data.

One, poll respondents sometimes choose issue positions based on their partisan identification (as opposed to holding an independent, a priori position). An example of this is a November 2015 poll[1] asking respondents whether the unemployment rate had increased or decreased under President Obama. A bare majority, 53% of Republicans said it had increased, while 76% of Democrats correctly answered that it had decreased. The opinions of Independents were not provided.

A related caveat is that partisan positioning may have shifted over time, particularly following the 2016 presidential election.

Two, these are national polls and thus cannot be used to divine partisan issue divides in specific states or Congressional Districts.

Three, issue preference distinctions between parties may mask key distinctions within parties, such as on abortion.

Issues are presented in no particular order. If polls were conducted across multiple months, I use the last month the poll was in the field.

**********

Income inequality. Democrats (87%, vs. 11% opposed, +76) and Independents (60%, +31) felt in 2015[2] that wealth is not fairly distributed among Americans, that the federal government should seek remedies (D 81%, +66; I 54%, +13), including increasing taxes on the wealthy[3] (D 84%, +72; I 63%, +32); smaller majorities of Republicans do not see this inequality (51%, +9) and oppose governmental remedies (64%, +30), including higher taxes (55%, +17).

Environment. Democrats and Independents strongly support the Paris] Agreement, oppose federal support for coal mining, believe climate change is man-made and support government intervention to reverse it. Smaller majorities of Republicans are more skeptical of climate change, support economic growth over environmental protection and oppose government intervention to reverse climate change/reduce global warming.

Reality of climate change. Democrats (80%, +61 relative to “About the same”) and Independents (54%, +13) felt in April 2017[4] that there had been “More extreme or unusual weather in the United States” in the past few years, while Republicans (60%, +33) thought it had been “About the same.”

Government role in fighting climate change. An April 2017 poll[5] found that 91% (+84) of Democrats and 73% of Independents (+49) were OPPOSED to “significantly cutting for scientific research on the environment and climate change, while 50% (+5%) of Republicans felt the opposite.

When asked in September 2014 which should receive higher priority, environmental protection or economic growth, 63% of both Democrats (+29) and Independents (+32) prioritized the environment, while a bare majority of Republicans (51%, +11) chose economic growth.

Coal production and fossil fuels. An April 2017 poll[6] found a deep partisan divide, perhaps driven by President Trump’s vociferous support for coal miners, over whether the federal government should encourage or discourage coal production. Democrats (80%, +66) and Independents (58%, +23), seeking to protect the environment, strongly favored “discourage,” while Republicans (69%, +50), seeking to protect coal jobs and the economy, strongly favored “encourage.” Still, this is not likely to be a winning issue for Democrats in coal-producing states such as West Virginia, which Trump won by 41.7 percentage points.

When asked in April 2016[7], 71% (+48) and 56% (+21) of Independents thought it would be a good idea for colleges and universities to stop investing in fossil fuels to reduce “global warming.” A majority of Republicans (55%, +18) disagreed.

Paris Agreement. In December 2015[8], Democrats were overwhelmingly in favor (86%, +77) of the United States joining “an international treaty requiring America to reduce emissions in an effort to fight global warming,” with Independents only slightly less enthusiastic (66%, +41). Republicans, while opposed, were far more evenly divided (52%, +10).

However, in what could be an electoral artifact, by June 2017 a clear majority of Republicans (68%, +47; averaging three polls[9]) supported President Trump’s announced withdrawal from the Paris Agreement, while 85% (+78) of Democrats and 62% (+33%) of Independents were opposed.

Planned Parenthood. In a September 2015 poll[10], 82% (+70) of Democrats and 56% (+19) of Independents supported federal government support for Planned Parenthood. Technically, there were “opposed to cutting off” these funds. Republicans (71%, +46) preferred to cut off the funds.

Gun control/rights. There is near-unanimity across all partisan groups for universal background checks and preventing terrorists from acquiring guns, although when either issue is framed as something supported by President Obama, Republican support plummets. For some additional context, please see this post.

Gun sales. Between July 2015 and April 2017, CBS News asked[11] six times whether gun sales should be made more strict, less strict or kept as they are; for ease of presentation, I combined response for “less strict” and “kept as they are.” On average, 78% (+58) of Democrats thought gun laws should be made more strict; only 51% (+6) of Independents concurred. A solid 64% (+30) of Republicans, meanwhile, felt that gun laws should either be kept as they are (48%) or made less strict (16%). These results echo a June 2017 poll[12] in which 80% (+62) of Democrats and 54% (+12) of Independents support stricter gun control laws, with 68% (+41) of Republicans opposed.

Personal safety. Two polls (July 2016,[13] June 2017[14]), asked whether more guns or fewer guns would make the United States safer. Allowing for slight question wording differences: an average 82% (+70) of Democrats 50% (+10) of Independents said more guns would NOT make us safer; an average 70% (+48) of Republicans felt the opposite.

Perhaps reflecting the geographic self-segregation of Democrats into more urban areas and Republicans into more exurban and rural areas, Democrats (77%, +62) and Independents (64%, +36) in November 2015 said[15] they were more worried about being the victim of gun violence, while Republicans were more worried (barely: 50%, +5) about a terrorist attack.

Majorities of Democrats (82%, +69) and Independents (57%, +21) in October 2015[16] thought “better gun regulation” would reduce mass shooting, while 59% (+28) of Republicans favored “more people carrying guns.” Similarly, that some month[17], Democrats (79%, +60) and Independents (55%, +17) were opposed to “allowing more teachers and school officials to carry guns in schools,” while Republicans (64%, +30) were in favor.

Health care. Perhaps no issue divides Democrats and Independents from Republicans more than the 2010 Patient Protection and Affordable Care Act, more colloquially known as the ACA or Obamacare, despite widespread (if barely among Republicans) agreement that Americans with pre-existing conditions should not be charged more for their health insurance nor should Medicaid enrollment to pre-2010 levels[18]. There was also partisan accord, in February 2015[19], on requiring “parents to vaccinate their children for diseases like measles, mumps, and rubella.”

ACA. A series of polls[20] conducted between May and August 2017 found that nearly all Democrats (85-91%, +77-87) and most Independents (57-65%, +28-41) opposed Republican ideas to repeal-and-replace Obamacare, while Republicans (58-61%, +24-34) mostly favored these ideas.

At the same time, in April and July 2017[21], on average, Democrats (90%, +83) and Independents (69%, +42) overwhelmingly supported making improvements to the ACA. By contrast, Republicans favored (65%, +34) continuing repeal-and-replace efforts. An April 2017 poll[22], conducted before the House approved the American Health Care Act, echoed this sentiment.

Obamacare exchanges. In June 2015[23], “the Supreme Court ruled that government assistance for lower-income Americans buying health insurance through both state-operated and federally-operated health insurance exchanges is legal.” Two polls conducted that month[24] found that, on average, Democrats (84%, +72) and Independents (67%, +20) favored this governmental largesse, while Republicans were (barely) opposed (52%, +10).

Single payer/Medicare-for-all. An series of polls in June and August 2017 poll[25] found that, on average, most Democrats (75%, +59) and a majority of Independents (56%, +22) favored the expansion of Medicare to cover all Americans, with 60% (+29) of Republicans opposed.

Marijuana legalization. Three polls conducted between January 2014 and August 2017[26] found a strong partisan divide: on average, 63% (+31) of Democrats and 60% of Independents favored legalization, while 61% (+26) of Republicans were opposed. Interestingly, between 2014 and 2017, overall support for marijuana legalization increased from 51% (+7) to 61% (+28).

Immigration. This is an issue where it is difficult to separate support/opposition for/to President Trump from support/opposition for/to issues associated with him.

Border wall along southern United States border. In February 2017[27], this central tenet of Trump’s presidential campaign was opposed by most Democrats (87%, +70) and Independents (61%, +25) and just as strongly favored by Republicans (77%, +57).

Executive order suspending entrance from seven majority-Muslim nations for 90 days. Trump’s executive order induced a stark partisan divide in early February 2017[28] (Democrats split 88-9% opposed, Republicans split 88-11% in favor), with Independents (51%, +6) just barely in opposition. A similar partisan divide was apparent on the question of a 120-day suspension of all refugee immigration: fully 92% (+84) of Democrats and 64% (+33) of Independents were opposed and 75% (+53) of Republicans were in favor.

When asked in September 2016[29] whether one supported or opposed “a blanket ban on the immigration of any person who lives in a country where there has been a history of terrorism against the west,” 78% (+62) of Democrats and 61% (+32) of Independents were opposed, while 54% (+16) of Republicans supported the idea.

In polls conducted in December 2015[30] and July 2016[31], an average 80% (+64) of  Democrats and 62% (+32) of Independents were opposed to a general Muslim ban, while Republicans (54%, +14) were somewhat in favor[32]. Other polls[33] released during this same time period had similar findings.

Illegal immigration from Mexico. In April 2016[34], when asked whether “you feel your own personal way of life is or is not under threat from illegal immigrants from Mexico,” fully 89% (+80) of Democrats and 68% (+38) of Independents, while Republicans were more evenly divided, 51-46%.

Syrian refugees. In November 2015[35], there was broad agreement (78-15% overall) that Syrian refugees should “go through a stricter security clearance process than they do now.” However, there was a partisan divide on the more general question of Syrian refugees coming to the United States. On average across two polls (November and December 2015[36]), Democrats (64%, +30) and (barely) Independents (50%, +4) were in favor (with the security clearance caveat cited above) and Republicans (72%, +48) were not in favor. And in September 2015[37], Democrats (69%, +40) and Independents (51%, +8) [38] favored increasing the number of Syrian refugees, while Republicans were opposed (67%, +37).

Islam. In February 2017[39], respondents were asked, “Generally speaking, do you think the Islamic religion encourages violence more than other religions around the world, less than other religions around the world, or about the same as other religions around the world?” Relative to “More,” Democrats (66%, +52) and Independents (53%, +25) chose “About the same;” relative to “About the same,” Republicans (63%, +38) chose “More.”

Use of military force. Going back a few years, a September 2014 poll found that more Democrats (59%, +23) and Independents (57%, +19) described themselves as “doves” (the United States should rarely or never use military force) and more Republicans (69%, +44) described themselves as “hawks” (military force should be used frequently to promote United States policy).

Equal protection under the law. I include here all LGBT and race/ethnicity questions.

Transgender. There is widespread agreement (75-23% overall, according to an April 2016 poll[40]) with “laws that guarantee equal protection for transgender people in jobs, housing and public accommodations.” However, that agreement does not extend to military service. In reaction to President Trump’s tweets about the subject, an August poll[41] found that Democrats (91%, +84) and Independents (72%, +49) strongly favored allowing transgender people to serve in the military, while Republicans (60%, +28) were opposed.

Same sex marriage. In three polls conducted between June and October 2015[42], an average of 67% (+41) of Democrats and 59% (+29) of Independents felt same sex marriage should be legal, with 56% (+20) of Republicans feeling it should not be legal (despite the June 2015 Supreme Court ruling, Obergefell v. Hodges, legalizing same sex marriage in all 50 states).

Religious exemptions. Two April 2015 polls[43] queried the right of businesses to refuse service to LGBT customers on religious grounds, potentially violating anti-discrimination laws. The question wording was slightly different, but on average Democrats (74%, +52) and Independents (60%, 25%) opposed these exemptions and Republicans (62%, +34) supported them.

Voting Rights Act. A February 2015 poll[44] asked whether the Voting Rights Act (VRA) was still necessary “to make sure that blacks are allowed to vote.” There were two interesting divides on this question. First, Democrats (62%, +24) and Independents (52%, +5) thought the VRA was still necessary, while Republicans (61%, +24) did not. Second, and perhaps more telling, fully 76% (+53) of black respondents thought the VRA was still necessary, while white respondents split 48-50 against.

**********

In sum, majorities of rank-and-file Democrats and Independents (in opposition to majorities of rank-and-file Republicans)…

…believe wealth is not fairly distributed among Americans and the federal government should seek remedies, including increasing taxes on the wealthy.

…support the Paris Agreement, oppose federal support for coal mining and believe climate change is both real and man-made (with government intervention required to reverse it).

…strongly support federal funding of Planned Parenthood

…feel gun laws should be more strict, more guns will not make us safer (Independents more evenly divided), more worried about being the victim of gun violence than of terrorism, better gun regulation would reduce mass shooting (not more people carrying guns) and oppose allowing more teachers and school officials to carry guns in schools.

…strongly opposed efforts to repeal-and-replace Obamacare (opting overwhelmingly to fix the law), supported government assistance for lower-income Americans buying health insurance through state and federal health insurance exchange and favored a Medicare-for-all/single payer health insurance system.

…support the legalization of marijuana

…oppose building a wall on the southern border of the United States and any form of “Muslim ban,” feel that illegal immigration from Mexico has not hurt their way of life and support Syrian refugees entering the United States (under stricter security clearance)

…have (relatively) more benign views of Islam

…feel the United States should rarely or never use military force to promote policy.

…support allowing transgender people to serve in the military and same sex marriage, while opposing allowing businesses to refuse service to LGBT persons on religious grounds.

…and believe the Voting Rights Act is still necessary to protect ballot access.

Let the campaigns begin!

Until next time…

[1] Bloomberg Politics Poll conducted by Selzer & Company. Nov. 15-17, 2015. N=1,002 adults nationwide. Margin of error ± 3.1.

[2] CBS News Poll. July 29-Aug. 2, 2015. N=1,252 adults nationwide. Margin of error ± 3.

[3] CBS News/New York Times Poll. Nov. 6-10, 2015. N=1,495 adults nationwide. Margin of error ± 3.

[4] Quinnipiac University. March 30-April 3, 2017. N=1,171 registered voters nationwide. Margin of error ± 2.9

[5] Quinnipiac University. March 30-April 3, 2017. N=1,171 registered voters nationwide. Margin of error ± 2.9.

[6] Quinnipiac University. March 30-April 3, 2017. N=1,171 registered voters nationwide. Margin of error ± 2.9.

[7] 60 Minutes/Vanity Fair Poll. March 30-April 3, 2016. N=1,010 adults nationwide. Margin of error ± 4.

[8] Quinnipiac University. Dec. 16-20, 2015. N=1,140 registered voters nationwide. Margin of error ± 2.9.

[9] NPR/PBS NewsHour/Marist Poll. June 21-25, 2017. N=995 registered voters nationwide. Margin of error ± 3.1; Quinnipiac University. May 31-June 6, 2017. N=1,361 registered voters nationwide. Margin of error ± 3.2; ABC News/Washington Post Poll. June 2-4, 2017. N=527 adults nationwide. Margin of error ± 5

[10] Quinnipiac University. Sept. 17-21, 2015. N=1,574 registered voters nationwide. Margin of error ± 2.5.

[11] CBS News Poll. April 21-24, 2017. N=1,214 adults nationwide. Margin of error ± 3.

[12] Quinnipiac University. June 22-27, 2017. N=1,212 registered voters nationwide. Margin of error ± 3.4.

[13] McClatchy-Marist Poll. July 5-9, 2016. N=1,053 registered voters nationwide. Margin of error ± 3.

[14] Quinnipiac University. June 22-27, 2017. N=1,212 registered voters nationwide. Margin of error ± 3.4.

[15] McClatchy-Marist Poll. Oct. 29-Nov. 4, 2015. N=1,465 adults nationwide (margin of error ± 2.6), including 1,080 registered voters (± 3).

[16] Fairleigh Dickinson University’s PublicMind. Oct. 1-5, 2015. N=771 adults nationwide. Margin of error ± 4.2.

[17] CBS News/New York Times Poll. Oct. 21-25, 2015. N=1,289 adults nationwide. Margin of error ± 4.

[18] Politico/Harvard T.H. Chan School of Public Health. June 14-18, 2017. N=501 adults nationwide. Margin of error ± 5.3.

[19] CBS News Poll. Feb. 13-17, 2015. N=1,006 adults nationwide. Margin of error ± 3.

[20] Quinnipiac University. July 27-Aug. 1, 2017. N=1,125 registered voters nationwide. Margin of error ± 3.4; Kaiser Family Foundation. July 5-10, 2017. N=1,183 adults nationwide. Margin of error ± 3.

[21] ABC News/Washington Post Poll. April 17-20, 2017. N=1,004 adults nationwide. Margin of error ± 3.5.Margin of error ± 3.4; Kaiser Family Foundation. July 5-10, 2017. N=1,183 adults nationwide. Margin of error ± 3

[22] Quinnipiac University. April 12-18, 2017. N=1,062 registered voters nationwide. Margin of error ± 3.

[23] CNN/ORC Poll. June 26-28, 2015. N=1,017 adults nationwide. Margin of error ± 3.

[24] CNN/ORC Poll. June 26-28, 2015. N=1,017 adults nationwide. Margin of error ± 3; CBS News/New York Times Poll. June 10-14, 2015. N=1,007 adults nationwide. Margin of error ± 3.

[25] Quinnipiac University. July 27-Aug. 1, 2017. N=1,125 registered voters nationwide. Margin of error ± 3.4; Quinnipiac University. June 22-27, 2017. N=1,212 registered voters nationwide. Margin of error ± 3.4.

[26] CBS News Poll. April 8-12, 2015. N=1,012 adults nationwide. Margin of error ± 3.; Quinnipiac University. July 27-Aug. 1, 2017. N=1,125 registered voters nationwide. Margin of error ± 3.4.

[27] CBS News Poll. Feb. 17-21, 2017. N=1,280 adults nationwide. Margin of error ± 3.

[28] Quinnipiac University. Feb. 2-6, 2017. N=1,155 registered voters nationwide. Margin of error ± 2.9.

[29] Monmouth University Poll. Sept. 22-25, 2016. N=802 registered voters nationwide. Margin of error ± 3.5.

[30] Quinnipiac University. Dec. 16-20, 2015. N=1,140 registered voters nationwide. Margin of error ± 2.9.

[31] CBS News/New York Times Poll. July 8-12, 2016. N=1,358 registered voters nationwide. Margin of error ± 3.

[32] By September 2016, opposition to the proposed Muslim ban had ceased to be a partisan issue. A Monmouth University Poll (Sept. 22-25, 2016. N=802 registered voters nationwide. Margin of error ± 3.5) found Republicans opposed 54-32%.

[33] McClatchy-Marist Poll. July 5-9, 2016. N=1,053 registered voters nationwide. Margin of error ± 3;  Quinnipiac University. June 21-27, 2016. N=1,610 registered voters nationwide. Margin of error ± 2.4; ABC News/Washington Post Poll. Dec. 10-13, 2015. N=1,002 adults nationwide. Margin of error ± 3.5. Across these three polls, average Democratic opposition was 81%, average Independent opposition was 59% and average Republican support was 64%.

[34] Monmouth University Poll. Aug. 4-7, 2016. N=803 registered voters nationwide. Margin of error ± 3.5.

[35] CBS News Poll. Nov. 19-22, 2015. N=1,205 adults nationwide. Margin of error ± 3.

[36] CBS News/New York Times Poll. Dec. 4-8, 2015. N=1,275 adults nationwide. Margin of error ± 3.

[37] Pew Research Center. Sept. 22-27, 2015. N=1,502 adults nationwide. Margin of error ± 2.9.

[38] Interestingly, a majority of Independents (58%, +21) in a November 2015 Gallup poll (Nov. 20-21, 2015. N=1,013 adults nationwide. Margin of error ± 4) were opposed to the specific number of 10,000 or more refugees proposed.

[39] CBS News Poll. Feb. 1-2, 2017. N=1,019 adults nationwide. Margin of error ± 4.

[40] CNN/ORC Poll. April 28-May 1, 2016. N=1,001 adults nationwide. Margin of error ± 3.

[41] Quinnipiac University. July 27-Aug. 1, 2017. N=1,125 registered voters nationwide. Margin of error ± 3.4.

[42] CBS News/New York Times Poll. Oct. 21-25, 2015. N=1,289 adults nationwide. Margin of error ± 4.

[43] Quinnipiac University. April 16-21, 2015. N=1,353 registered voters nationwide. Margin of error ± 2.7; CNN/ORC Poll. April 16-19, 2015. N=1,018 adults nationwide. Margin of error ± 3.

[44] CNN/ORC Poll. Feb. 12-15, 2015. N=1,027 adults nationwide (margin of error ± 3), including 733 non-Hispanic whites (± 3.5), and, with an oversample, 309 blacks (± 5.5).

Clinton Derangement, or The Birth of a Notion

During the 2016 campaign, I was struck by two overlapping narratives.

One was the intensity of animus toward Democratic presidential nominee Hillary Clinton, particularly among Republicans. This felt to me less like policy disagreement and more like personal vendetta. This animus expressed itself manufactured outrage (and HOURS of Congressional hearings) over the Islamist militant attack on the U.S. diplomatic mission in Benghazi, Libya on September 11, 2012 and the use of a private e-mail server while serving as President Barack Obama’s first Secretary of State. While neither Benghazi nor the e-mail server cast Ms. Clinton in the best possible light, the obsessive focus on them as somehow criminal acts (leading to chants of “Lock her up!” at the 2016 Republican National Convention) smacked of political overkill. Thus, the adjective “manufactured.”

To me, this animus echoed the fervor with which Republicans had gone after her husband, Bill Clinton, while he served as president of the United States. The Whitewater-Lewinsky investigations, which ran (with brief stoppages due to lack of evidence) from January 1994 to Clinton’s acquittal by the United States Senate (Senate) on February 12, 1999, were an earlier manifestation of what I will call “Clinton derangement.”

Two was the emergence, led by Donald Trump, of a virulent strain of raw nationalist populism, driven by mass rallies, calls for violence against journalists and protestors, and anti-elitist resentments (aimed at the establishments of both major political parties). Trump encouraged this nationalist populism through outrageous, often vulgar statements and Twitter musings, culminating in the Access Hollywood audiotape, released on October 7, 2016, on which Trump appears to brag about sexually assaulting women.

So, you ask, where is the overlap?

Well, consider the curious case of U.S. House of Representatives (House) member Jason Chaffetz (R-UT).

Immediately after the release of the Access Hollywood audiotape, Rep. Chaffetz announced that he was no longer supporting Trump’s presidential candidacy. His reason, passionately expressed during a media blitz, was that he could not explain to his 15-year-old daughter how he could support someone who utters “some of the most abhorrent and offensive comments that you can possibly imagine.”

Just 19 days later, however, Rep. Chaffetz tweeted this: “I will not defend or endorse @realDonaldTrump, but I am voting for him. HRC is that bad. HRC is bad for the USA.”

I wonder how that second conversation between Rep. Chaffetz and his 15-year-old daughter went.

In short: for Republicans, no matter how “abhorrent” (Rep. Chaffetz’s word) or “reprehensible” (the word of a Republican cousin-by-marriage who still voted for Trump) Trump’s behavior or how far he strayed from Republican orthodoxy, Hillary Clinton was somehow simply worse.

I wrote “strayed from Republican orthodoxy,” but the reality is that the nationalist populism embraced by Trump already existed among a significant segment of Republican voters.

Simply put, Trump did not create his voters, his voters created him.

**********

The more I thought about these two strands, Clinton derangement and an evolving Republican strain of nationalist populism, the more I found myself thinking about the presidential election of 1992 (and the subsequent midterm elections of 1994).

And an argument began to take shape in my mind, one that greatly clarifies both Trump’s emergence and Clinton’s defeat:

  1. With the elections of 1992 and 1994, the Democratic and Republican Parties switched governing roles. The Democratic Party went from being primarily a Congressional and state-house party to primarily a national (i.e., White House) party, while the Republican Party went in the opposite direction.
  2. Republicans blamed Bill Clinton for breaking their iron grip on the White House, and they have been punishing him (and his wife) for it ever since.
  3. The Republican Party underwent a profound transformation in the aftermath of the first Bush presidency (1989-93) as a result of both internal actions (e.g., Bush breaking his “read my lips” pledge not to raise taxes) and external events (e.g., the end of the Cold War). The common enemy (the Soviet Union) which had held together the disparate ideologies of the Republican Party for 25 or 30 years vanished, and previously dormant “fringe” factions asserted themselves.
  4. This newly radical Republican Party was characterized in part by a nationalist populism (embodied by Pat Buchanan) and a resentful anti-government ethos (embodied by Georgia Republican Congressman Newt Gingrich). And they had a new common enemy: the Clintons.

I will explore these arguments over a series of posts, beginning with this one.

*********

Just bear with me while I present three election data “items” related to the 1992 and 1994 elections.

Item 1.[1]

On Tuesday, November 3, 1992, Clinton captured 43.0% of the popular vote cast for president, 5.6 percentage points more than G. H. W. Bush (37.4%) and 24.0 percentage points more than Independent H. Ross Perot (19.0%). Considering only votes cast for the Democratic and Republican presidential candidates (“two-party vote”), Clinton beat Bush 53.5 to 46.5%.

Clinton also won 32 states, plus the District of Columbia (DC), for a total of 370 electoral votes (EV); Bush received only 168 EV.

electoral-1992-s

Four years earlier, however, G. H. W. Bush had won 53.4% of the popular vote against Democrat Michael Dukakis (45.6%), winning 40 states and 426 EV; Bush beat Dukakis 53.9 to 46.1% in the two-party vote. Bush’s near-landslide victory (in the Electoral College, anyway) meant that Republicans would control the White House for a third consecutive four-year term.

electoral-1988-sIn other words, in just four years, Democrats had increased their share of the two-party popular vote by 7.4 percentage points, flipped 22 states from Democratic to Republican, and increased their EV total from 112 to 370.

That is an astonishing turnaround.

Item 2.

Following the 1992 elections, Democrats controlled 57 Senate seats and 258 House seats, while 30 of the nation’s governors were Democrats.

Just two years later, following the 1994 midterm elections, Democrats controlled 48 Senate seats and 204 House seats; only 19 governors were Democrats.

Put differently, just two years after Democrats regained control of the White House for the first time in 12 years, Democrats were absolutely slaughtered in Congressional and gubernatorial races, losing a net nine Senate seats, 54 House seats and 11 governor’s mansions.

That is not a turnaround. That is whiplash.

Item 3a.

Table 1: Summary of Democratic performance in Presidential, Congressional and gubernatorial races: 1968-2016

Presidency 1968 – 1988

(24 years)

1992 – 2016

(28 years*)

Post-1988 Change
Average % Total Vote 42.9% 48.7% D+5.8 perc pts
Average % 2-Party Vote 45.1% 52.0% D+6.9 perc pts
Average #Electoral Votes 113.0 313.4 D+200.4 EV
Average States Won 9.0 (+DC) 23.7 (+DC) D+14.7 states
Victories 1 4†† D+3 wins
Years in White House 4 16 D+12 years
 
Senate 1968-1992

(26 years)

1994-2016

(24 years)

Post-1992 Change
Average % Total Vote 51.2% 48.8% D-2.4 perc pts
Average % 2-Party Vote 53.0% 51.2% D-1.8 perc pts
Average # Seats 54.5 48.3 D-6.2 seats
Years in Majority 20 9.5 D-10.5 years
House
Average % Total Vote 52.9% 47.4% D-5.5 perc pts
Average % 2-Party Vote 54.1% 49.6% D-4.5 perc pts
Average # Seats 262.1 208.7 D-53.4 seats
Years in Majority 26 4 D-22 years
Governor
Average % Total Vote 51.7% 46.7% D-5.0 perc pts
Average % 2-Party Vote 52.9% 48.8% D-3.9 perc pts
Average # State Houses 31.0 20.7 D-10.3 state houses
Years in Majority 26 4 D-22 years

         *Through 2020

         Through 2018

         ††Excluding popular vote victories by Al Gore in 2000 and Hillary Clinton in 2016

In the six presidential elections from 1968 through 1988 (Table 1), Republicans won the presidency five times, four times by landslides (1972, 1980, 1984) or near-landslides (1988). The one Democratic victor was Jimmy Carter in 1976, in the wake of Republican President Richard Nixon’s Watergate-related resignation in August 1974, Nixon’s pardon by his successor (Gerald Ford) and various Ford gaffes. Still, Carter only managed to beat Ford by 2.1 percentage points (50.1 to 48.0%) and 57 EV (297-240[2]); Ford actually won more states: 27 to 23 (plus DC). In fact, had Ford flipped 5,559 votes in Ohio (25 EV) and 7,232 votes in Mississippi (7 EV)—just 12,791 votes out of 81,540,780 cast, he would have won 272 EV and held on to the presidency.

Overall in those six presidential elections, the Democratic candidates averaged 42.9% of the popular vote (45.1% of the two-party vote), victories in nine states (plus DC) and 113.0 EV. The White House essentially “belonged” to the Republicans during this period.

During the same time period, however, Democrats controlled the House and held a majority of governorships. They controlled the Senate for 18 of 24 years, excepting only 1981-87. Following the 13 even-numbered elections from 1968 through 1992, Democrats averaged majorities of all votes cast for Senate, House and governor[3], for an average of 54.5 Senate seats, 262.1 House seats and 31.0 governor’s mansions.

In other words, from 1968 through 1992, while Republicans held a near lock on the White House, Democrats controlled Congress (both Houses for 20 years) and a majority of governor’s mansions. One interpretation is that voters preferred Republicans in the White House to conduct foreign policy (i.e., fight the Cold War) and preferred Democrats to manage domestic affairs (i.e., protect entitlements).

Item 3b.

AFTER 1992, however…

In the seven presidential elections from 1992 through 2016, Democrats won the presidential popular vote six of seven times (despite only winning the Electoral College—and thus the White House—four times), the exception being 2004, when Republican George W. Bush won reelection by 2.4 percentage points (50.7 to 48.3%) over Democrat John Kerry, capturing 286 EV to Kerry’s 251.[4] Once again, Ohio was pivotal. If Kerry could have flipped 59,188 votes there (out of 122,303,590 cast nationwide), he would have won 271 EV and the presidency (though he still would have lost the popular vote).

Overall in those seven presidential elections, the Democratic candidates averaged 48.7% of the popular vote (52.0% of the two-party vote), victories in 23.7 states (plus DC) and 313.4 EV.

Meanwhile, since January 1995, Democrats have only controlled the House and held a majority of governorships for four years (2007-11), while controlling the Senate for only nine-plus years (May 2001[5]-January 2003, 2007-15). Following the 12 even-numbered elections from 1994 through 2016, while Democrats managed rough parity in Senate votes, they lost the overall vote for House and governor, earning an average 48.3 Senate seats, 208.7 House seats and 20.7 governor’s mansions.

The overall lower average of EV won by White House winners and the near-parity in Senate votes means that the more recent differentiation between Democrats as “White House” party and Republicans as “Congressional/gubernatorial” party is not quite as apparent.

Figure 1. Democratic % Presidential Popular Vote, U.S. Senate Vote, U.S. House Vote, Governor Vote: 1968-2016

Democratic % President, Senate, House, Governor, 2-party, 1968-2016

Figure 2. Democratic % Electoral Votes, U.S. House Seats, U.S. Senate Seats, Governorships: 1968-2016

Democratic % EV, Senate Seats, House Seats, Governors, 1968-2016

The Democratic presidential victory in 1976, Republican presidential victories in 2000 and 2004, and Democratic successes across the board in 2006 and 2008 obscure the trends in Figure 1 (Democratic percentages of the two-party vote cast for president, Senate, House and governor, 1968 through 2016) and Figure 2 (Democratic percentages of EV, Senate and House seats, and governorships, 1968 through 2016). However, the dramatic shift in governing roles (“national” vs. “legislative”) can still be discerned, especially in Figure 2, which focuses on the winning of control-determining EV, Congressional seats and governor’s mansions.

Figures 3 and 4 clarify the 1992-94 governing role shift by displaying, for each presidential election, the average of the Democratic results in that election and the previous one, and for each set of Senate, House and (two-year cycle) gubernatorial elections, the average of the Democratic results in that election and the previous two elections. In other words, each value in Figures 3 and 4 encompasses an identical rolling three-even-year-elections time frame.

Figure 3. Democratic % Presidential Popular Vote, U.S. Senate Vote, U.S. House Vote, Governor Vote: 6-Year Rolling Averages, 1968-2016

Democratic % President, Senate, House, Governor, 2-party, 6-yr-ave, 1968-2016

Figure 4. Democratic % Electoral Votes, U.S. Senate Seats, U.S. House Seats,      Governorships: 6-Year Rolling Averages, 1968-2016

Democratic % EV, Senate Seats, House Seats, Governors, 6-yr-ave, 1968-2016

The dramatic shift in party governing roles in 1992-94 is now strikingly clear, especially in Figure 4. What is also readily apparent is just how poorly Democrats have been faring in governor’s races since 1992. I have written elsewhere about the current extreme Republican advantage in governor’s mansions (16 Democratic, 33 Republican, 1 Independent). As Figures 2 and 4 show, the gap was not very different in the mid-to-late 1990s, during which Democrats held an average of only 18 governor’s mansions.

In a subsequent post, I will examine the defining events of 1988 through 1994 in more detail, moving from then-Vice-President G. H. W. Bush’s acceptance speech at the 1988 Republican National Convention through the wildly successful (for Republicans) 1994 midterm elections. I will follow that post with one in which I theorize about how resentment at Bill Clinton’s 1992 (and 1996) electoral successes and the Republican Party’s shift from “national” to “legislative” party may have resulted in the “surprising” 2016 elections.

Until next time…

[1] All presidential and Congressional election are from Dave Leip’s indispensable Atlas of U.S. Presidential Elections, as are 1990-2016 gubernatorial election data. Gubernatorial data from 1967 to 1989 from the equally invaluable OurCampaigns.

[2] Ronald Reagan, who had challenged Ford for the Republican nomination, won the remaining EV, cast by a Washington state elector.

[3] Five states still hold gubernatorial elections in odd-numbered years: Kentucky, Louisiana and Mississippi (next election: 2019), and New Jersey and Virginia (next election: 2017). To maintain consistency with presidential and Congressional elections, I combined the election results for gubernatorial elections held in even-numbered years with those held in the preceding odd-numbered year.

[4] Democratic Vice-Presidential nominee John Edwards won the remaining EV, cast by a Minnesota elector.

[5] On May 24, 2001, Republican Senator Jim Jeffords of Vermont switched to “Independent,” turning a 50-50 Senate tie, broken by Republican Vice President Dick Cheney into a 50-49 Democratic edge. In the subsequent 2002 midterms, however, Democrats lost a net of two seats, giving the Republicans a 51-48-1 edge.

The AHCA vote likely increased Democrats’ chances of winning the U.S. House in 2018

Alaska and Hawaii became the 49th and 50th states, respectively, in 1959. As a result, 1962 was the first election year to reflect the current U.S. House of Representatives (House) configuration of 435 seats allocated across 50 states[1]. These were also the first House elections since John F. Kennedy won the presidency two years earlier. The rule of thumb is that the party of the president loses House seats in the first midterm (even-numbered years between presidential election years) elections following the first time that president is elected.

In 1962, the Democrats lost five seats, the Republicans won two seats, there was one new Independent, and two seats were eliminated (the previous two Houses had 437 seats in anticipation of Alaska and Hawaii seats and the post-1960 U.S. Census redistricting). To simplify matters, let’s call this a Democratic loss of two House seats.

**********

On May 3, 2017, the House voted 217-213 to pass the American Health Care Act (AHCA). Of the 431 current House members, all 193 Democrats voted against the bill, as did 20 of the 238 Republicans, with one Republican (Dan Hewhouse of Washington’s 4th congressional district [WA-4]) not voting.

The AHCA was the third major attempt to overhaul the U.S. health care system in the last 25 years. On October 27, 1993, President Bill Clinton introduced his universal health care bill (designed in part by a task force led by First Lady Hillary Clinton) to Congress. After months of wrangling, it never came to a vote in either the House or the U.S. Senate (Senate), and the effort to pass the bill ended in late August 1994.

On September 9, 2009, President Barack Obama delivered a nationally-televised address to a joint session of Congress calling for Congress to pass health care reform, with the goal of universal coverage. Initial legislation passed the House on November 7, 2009 and the Senate on December 4, 2009. The Senate bill, the Patient Protection and Affordable Care Act (ACA), was amended by the House to allow the Senate to pass it on a simple majority vote under a process called budget reconciliation[2]. Ultimately the amended ACA passed both houses of Congress and was signed into law by President Obama on March 23, 2010.

Both of these months-long, highly-controversial efforts were undertaken by a newly-elected president in the two years prior to his first midterm elections.

Hold that thought.

**********

As noted earlier, it has been common for a president’s party to lose House seats in that president’s first midterm elections. There have been eight such elections since 1960 (I exclude 1966, since Lyndon Johnson won RE-election to the presidency in 1964, and 1974, since Gerald Ford had not been elected to the presidency):

Figure 1: Net House Seat Losses for a President in First-Term Midterm Elections, 1962-2010

Net House Seat Loss by President’s Party in First Midterm, 1962-2010

Starting in 1962, the president’s party has lost an average of 21.5 House seats (median loss=13.5 seats) in that president’s first midterm elections (Figure 1)[3]. However, the rate of first-midterm House seat loss has accelerated over time. A simple weighting scheme[4] increases the average to 27.4 House seats lost. The simple average over the last five qualifying midterm elections is 28.6 House seats lost (median=26), and over the last three is 36.3 House seats lost (median=54). Finally, a simple linear regression model “predicts” that the average House seat loss for Republicans in 2018 would be 45.9!

Clearly, history is not always predictive. The president’s party lost an average of 13.8 House seats in the four qualifying midterm elections from 1962-1982, yet President George H.W. Bush’s Republicans only lost 8 House seats in 1990, while President Bush was still receiving plaudits for the first Gulf War and the end of the Cold War. Similarly, the 1962-1994 average was 19.5 House seats lost, yet President George W. Bush’s Republicans actually GAINED 8 House seats in 2002, in the first midterm elections following the 9-11 terrorist attacks.

Still, here is where the examples of 1994 and 2010 are instructive. As noted earlier, these were the first midterm elections following a newly-elected president, with majorities in both the House and Senate, attempting to overhaul the nation’s health care system; both overhaul attempts were highly controversial and extended over much of the president’s first two years in office.

Democrats lost an average of 58.5 House seats over those two elections, handing control of the House to Republicans on both occasions (in 1994, for the first time in 40 years!)[5].

Could the same thing happen to Republicans in 2018, following the AHCA vote in the House, and what appears likely to be as long a process as the previous two attempts to change the U.S. health care system? Analysts I respect are already speculating about the possibility, but just bear with me while I offer my own analysis.

**********

In the 2016 midterm elections, Republicans won 241 House seats and Democrats won 194 House seats, meaning that Democrats would have to gain 24 House seats to win a majority of House seats in 2018. I have already written about the challenge to such a takeover presented by the large majority of very safe seats held by both political parties. In that earlier post, I found only 48 seats (28 won by Republicans[6]) decided by less than 10 percentage points and/or the winning candidate won less than 55% of the total vote (“close”); there were only 23 districts won by a Republican House candidate AND by Democratic presidential nominee Hillary Clinton. In total, there are 41 House districts in either/both of these two “vulnerable” categories.

The easiest thing for Democrats would be to win 24 of these 41 House seats in 2018 (while losing none of their seats). And, in an intriguing sign of things to come, 15-term Republican Congresswoman Ileana Ros-Lehtinen (FL-27) announced that she would not seek reelection in 2018. In 2016, Ms. Clinton won her House district, and she won her House seat by “only” 9.8 percentage points. Democrats are now favored to win this seat in 2018.

There are other warning signs for Republicans as well. After Republican Mike Pompeo won his House seat (KS-4) in 2016 by 31.1 percentage points, Republican Ron Estes won a special election for that seat on April 11, 2017 by just 6.8 percentage points, a pro-Democratic shift of 24.3 percentage points. Republican Tom Price won his House seat (GA-6) in 2016 by 23.4 percentage points, but Democratic candidate Jon Ossoff is essentially tied with Republican Karen Handel in advance of the June 20 runoff in the special election to fill that seat, a shift of 23.4 percentage points. A total of 93 Republicans won their House seats in 2016 by less than 23 percentage points; if Democrats won only one-third of these Republican seats in 2018, that would still be seven more than they need to recapture the House.

**********

It is remarkable political irony that the ACA is now more popular than ever. On April 1, 2010, just after the ACA was signed into law, 42.7% supported it and 50.0% opposed it. Seven years later, those numbers had reversed: 48.4% support and 41.9% opposition, a 13.8 percentage point marginal shift.

At the same time, the AHCA has been less popular than the ACA ever was. The public disapproves of Congress’ performance (62.9% disapprove, 17.2% approve) and of the way President Donald Trump has handled health care (51.3% disapprove, 37.7% approve).

Still, 217 of 238 House Republicans in a historically unpopular body, acting at the behest of an unpopular president, voted in favor of an unpopular bill whose purpose was to “repeal and replace” an increasingly-popular law.

This is exactly what the creation of very safe seats does to democracy—it creates a perverse incentive to be more afraid of primary challenges from party extremes than of the broader pool of general election voters.

Table 1: Distribution of House Votes for the AHCA on May 4, 2017 By Vulnerability of House Member

  Close, Clinton Won Close, Trump Won Not Close, Clinton Won Not Close, Trump Won TOTAL
No 5 1 4 10 20[7]
Yes 5 17 9 186 217
TOTAL 10 18 13 196 237
% No 50.0% 5.6% 30.8% 5.1% 8.4%
Ratio 9.80 1.09 6.03

Overall, according to Table 1, only 8.4% of Republican House members (excluding Newhouse’s non-vote) voted no on the AHCA, as did an even smaller 5.1% of the 196 Republicans from the safest districts (2016 House vote margin>10 percentage points and/or the winning percentage>55%[8]) AND Trump won the district). Surprisingly, only 1 Republican House member (5.6%) from a close district that Trump won (Brian Fitzpatrick [PA-8]) voted no on the AHCA, while the other 17 voted yes, implying they are more scared of Trump voters in those districts than of all other voters.

On the other hand, four Republican House members (30.8%) from districts that were not close, but which Clinton won—John Katko (NY-24), Ryan A. Costello (PA-6), Patrick Meehan (PA-7), Dave Reichert (WA-8)—voted no, while nine voted yes. And, of the 10 Republican House members whose 2016 election was close AND whose district was won by Clinton, half voted no (Mike Coffman [CO-6], Ros-Lehtinen, Leonard Lance [NJ-7], Will Hurd [TX-23], Barbara Comstock [VA-10]) and half voted yes.

Thus, compared to “safe” Republican House members, Republican House members from the least safe (close, Clinton) districts were 9.8 times more likely to vote no on the AHCA, while Republican House members from the not-close districts Clinton won were 6.0 times more likely to vote no.

Simply put, Republican House members from districts won by Hillary Clinton in 2016 (regardless of how “close” their own 2016 election was) were 7.7 times more likely to vote no on the AHCA than Republican House members from districts won by Donald Trump.

But that still leaves 31 Republican House members from districts which were close in 2016 and/or Clinton won in 2016 who voted YES on the AHCA:

Close AND Clinton won (n=5)

Jeff Denham              CA-10

Steve Knight             CA-25

Darrell Issa                CA-49

Carlos Curbelo         FL-26

Kevin Yoder               KS-3

Close and Trump won (n=17)

Martha Robey           AL-2

Don Young                AK-At large

Scott Tipton               CO-3

Brian Mast                 FL-18

Mike Bost                  IL-12

Trey Hollingsworth IN-9

Rod Blum                  IA-1

David Young             IA-3

Bruce Poliquin         ME-2

Jack Bergman            MI-1

Don Bacon                 NE-2

John J. Faso                NY-19

Claudia Tenney        NY-22

Tom Reed                  NY-23

Lloyd K. Smucker    PA-16

Mia Love                   UT-4

Not Close and Clinton won (n=9)

Martha E. McSalley              AZ-2

David Valadao                     CA-21

Ed Royce                               CA-39

Mimi Walters                        CA-45

Dana Rohrabacher               CA-48

Peter Roskam                       IL-6

Erik Paulsen                         MN-3

Tom Culberson                    TX-7

Pete Sessions                        TX-32

Put it this way: if Ossoff wins the runoff on June 20 and a Democrat wins Ros-Lehtinen’s seat, the Democrats would need to win just 22 remaining seats to recapture the House (assuming they lost none of their own 20 close seats). There are 31 seats where voters can truly make an impact by continuing to hold their Republican House members accountable for their AHCA “yes” vote (in clear opposition to the overall political “lean” of their constituents).

And that does not count 9 other Republican House members who voted no on the ACHA but are still potentially vulnerable.

I am not predicting that the Democrats will recapture the House in 2018. I am only saying that a VERY strong opportunity to do so, based on accelerating historical trends, the closeness of 41 (of 241) seats, and the demonstrated impact of tinkering with national health care, is there, IF the Democrats can successfully capitalize on all of these trends.

Until next time…

[1] There had been a total of 435 House seats since 1912, after Oklahoma (1907), Arizona and New Mexico (both 1912) became the 46th through 48th states.

[2] This became necessary after the Democrats lost their filibuster-proof 60-seat Senate majority following the death of Massachusetts Senator Edward Kennedy on August 25, 2009 and the eventual victory of Republican Scott Brown (January 19, 2010) to replace him.

[3] Using the 1966 and 1974 data—10 first midterm election for any president—increases the average net House seat loss to 26.8.

[4] Where the 1962 value is weighted “1,” the 1970 value is weighted “2,” and so forth up to the 2010 value weighted “8.” Adding 1966 and 1974 increases the weighted average to 34.5 net House seats lost.

[5] Let me be very clear. I am NOT saying that the pursuit of national healthcare reform legislation was the sole reason for the Democrat’s very high House seat loss in 1994 and 2010. I would argue that, in 1994, the House Post Office scandal and the omnibus budget bill of 1993 (with its large tax increases) also played a huge role, as did backlash to the first black president and cap-and-trade legislation (which passed the House but died in the Senate) in 2010.

[6] This is actually a correction to that post, as I had neglected to count one close Republican House seat.

[7] Overall, 13 no votes were cast by Republicans from states Clinton won in 2016 (CO, NJ, NY, PA, VA, WA), six were cast by Republicans from states Clinton lost by less than 10 percentage points (AZ, FL, NC, OH, TX) and the other was cast by Thomas Massie of the 4th CD of Kentucky, a state Clinton lost by 29.8 percentage points. Go figure.

[8] This represents 81.3% of all Republican House seats won in 2016.

Gerrymandering is a bigger problem for democracy than for Democrats

It is an article of faith among Democrats and some political commentators that a major barrier to Democrats retaking control of the House of Representative in 2018, or even in 2020, is Republican gerrymandering following the 2010 U.S. Census. Republicans, the narrative goes, used the governor’s mansions and state legislatures they controlled after the 2010 midterm elections to draw state legislative and U.S. House districts to their partisan advantage.

Gerrymandering is nearly as old as the Republic. The word “gerrymander” comes from Elbridge Gerry, the Massachusetts governor who supervised the redrawing of his state’s legislative districts (U.S. House, state senate, state house) to advantage his Democratic-Republicans following the 1810 U.S. Census. One new state senate district resembled a salamander, leading to the term “Gerry-mander” to describe the drawing of legislative district lines for partisan advantage.

However, outside of “compactness” and rough equality of population across districts of the same type in the same state (under the 14th Amendment’s Equal Protection Clause, per Reynolds v. Sims [1964]), there is no official guidance for drawing legislative districts. Article 1, Section 2 of the U.S. Constitution says only that Representatives “shall be apportioned among the several States…according to their respective Numbers…[and] each State shall have at Least one Representative,” thus requiring the United States Census Bureau to determine every 10 years how many U.S. House members each state has (and thus how many electoral votes it has [# U.S. House members plus 2]). And the Voting Rights Act mandates majority-minority districts under certain circumstances.

What does not exist is a measure of how “fair” legislative districts are. That is why, until very recently, courts have been reluctant to overturn legislative district maps that may proffer a partisan advantage, yet still adhere to the requirements of compactness, relative population equality and, where necessary, the creation of majority-minority districts.

Naturally, I will now describe and analyze two potential, related measures of redistricting “fairness,” utilizing two-party vote for the U.S. House of Representatives[1].

Turning votes into seats. The first measure is the seat-to-vote ratio (SVR):  the percentage of legislative districts (seats) won by a party in an election divided by the percentage of the vote won by that party. This is a straightforward concept—if one party wins 53% of the vote, you would expect that party to win (very close to) 53% of available seats. An SVR of 1.00 is a perfect correspondence of seats and votes, while an SVR greater than 1.00 suggests a partisan advantage.

Figure 1: Percentage Democratic U.S. House Seats and Two-Party U.S. House Vote, 1968-2016

 Dem % Seats and Votes, U.S. House, 1968-2016

In 2016, Democrats won 47.6% of the total U.S. House vote, and Republicans won 48.7%, equivalent to 49.4% and 50.6% of the two-party vote, respectively (Figure 1). However, Democrats “only” won 44.6% (194) of the seats, when they “should” have won 49.4% (~215 seats; SVR=0.90). In other words, Democrats won 10% fewer seats than you would expect if seats and votes lined up evenly.

Just bear with me while I dive into the arcana of counting U.S. House votes. Feel free to skip the next six italicized paragraphs.

My sources for U.S. House election data are the biennial Statistics of the Presidential and Congressional Election reports published by the Office of the Clerk, U.S. House of Representatives (House Clerk). These reports tabulate votes for the Democratic and Republican candidates as well as for every other smaller party candidate (e.g., Libertarian, Green) as well as No Party Affiliation/No Political Party/Nonaffiliated/Nonpartisan, None of these Candidates (Nevada), Nominated by Petition (Iowa), Write-in/Other Write-in votes, Blank Votes (Hawaii, Maine, Massachusetts, New York, Vermont), Under Votes (Wyoming), Spoiled Votes (Vermont), Void (New York), Over Votes (Hawaii, Wyoming), Scatter/Scattering (New Hampshire, Wisconsin), All Others (Massachusetts) and Miscellaneous (Oregon).

 California and Washington have all candidates for an office, regardless of party affiliation, run in a single “jungle primary” prior to Election Day. If no candidate receives more than 50% of the vote, the top two finishers face off on Election Day. In seven Congressional Districts (CD) in California and one CD in Washington, two Democrats faced off on Election Day, while one CD in Washington had two Republicans facing off on Election Day. I recorded all Democratic and Republican votes from these nine CD in my major party vote totals. Similarly, Louisiana holds their jungle primary on Election Day (with run-offs for the top two vote winners in December). I counted all Democratic and Republican votes on Election Day in my major party vote totals. Finally, Alabama, Georgia, Kentucky, North Carolina and South Dakota only list the Democratic and Republican candidates on their Election Day ballots.

Three states tabulate candidate votes from multiple party lines because smaller parties officially endorsed either the Democrat or Republican candidate for an office; the official House Clerk Democratic and Republican vote tallies exclude these minor party votes. Democrats in Connecticut received votes on the Working Families line, and Republicans received votes on the Independent line. In New York, Democrats received votes on the Working Families, Women’s Equality and, in CDs 2, 18 and 20, Independence lines, and Republicans received votes on the Conservative, Reform, Blue Lives Matter, Stop Iran Deal and, in CDs 1, 10, 11, 13, 19, 21, 23-25 and 27, the Independence line. In South Carolina, Democrats running in CDs 1, 2 and 7 received votes on the Working Families and Green lines. I included these smaller party votes in my major party vote totals because I also wanted to count the margin of votes separating the top two finishers in each U.S. House race.

In Minnesota, the Democratic Party is listed in CDs, 1, 4, 5, 7 and 8, while the Democratic-Farmer-Labor Party is listed in CDs 2, 3 and 6. Following Minnesota tradition, I counted all of these votes as “Democratic” in my major party votes totals.

Two states—Florida and Oklahoma—do not print unopposed candidates’ names on their ballots. This means that I have no votes for Democrat Frederica S. Wilson in Florida CD 24 and Republican Jim Bridenstine in Oklahoma CD 1.

For these reasons, while the House Clerk and I agree on a total of 129,888,395 votes cast in U.S. House elections in 2016, the House Clerk breakdown is 47.3% Democratic, 48.3% Republican, 4.3% All Other votes, while my breakdown is 47.6%, 48.7% and 3.7%–a difference that does not materially affect these analyses.

Figure 2: Ratio of Percentage Democratic U.S. House Seats and Percentage Democratic of Two-Party U.S. House Vote, 1968-2016

Dem Seat-Vote Ratios, U.S. House, 1968-2016

The Democrats’ 0.90 SVR in 2016 is in line with their 0.91 and 0.92 SVRs from 2012 and 2014 (Figure 2), suggesting that their disadvantage in converting votes to seats has been consistent since the 2010 redistricting.

However, as Figures 1 and 2 show, it is perhaps hypocritical for Democrats to yell too much about having SVR<1.00. Between 1968 and 1992, Democrats averaged 60.1% of U.S. House seats. They also averaged SVRs of 1.13 in the five elections following the 1970 redistricting (1972-80) and 1.11 in the five elections following the 1980 redistricting (1982-90), higher than the average 1.09 SVR for Republicans since 2010.

In 1994, the Republicans netted 54 U.S. House seats to win the majority for the first time since 1952, and they have held that majority for all but four years (2007-10) since. Still, in the five elections after each of the 1990 and 2000 redistricting processes (1992-2010), the U.S. House was, on average, evenly divided (50.1% Democratic) with an average SVR of 1.00.

In other words, when one party is consistently in the majority (Democrats in the 1970s and 1980s; Republicans since 2010), their SVR tends to be greater than 1.00. But is that a sign of systematic bias (partisan redistricting) or merely success at winning a majority of close elections?

One way to address this question is to examine results at the state level.

Seven states (Alaska, Delaware, Montana, North Dakota, South Dakota, Vermont, Wyoming) have only one U.S. House member, rendering gerrymandering moot. Four states (Arizona, California, Idaho and Virginia) use an independent commission for redistricting, ostensibly removing political motivations from redistricting[2].

Removing the 82 U.S. House seats in these 11 states leaves Democrats with 145 (33.8%) seats after the 2016 elections, and 47.7% of the two-party vote, for a SVR of 0.71. That is, in the 39 states where partisan gerrymandering could happen, Democrats won 29% fewer seats than would be expected based upon their two-party vote share. In these states, moreover, Republicans netted 21.7 “extra” seats (Table 1).

Table 1: 2016 U.S. House Election Results in 39 States Where State Legislatures Control Redistricting

State U.S. House Seats Democratic % 2-Party Vote Actual Democratic Seats Expected Democratic Seats Actual – Expected
Pennsylvania 18 45.9% 5 8.3 -3.3
Texas 36 39.3% 11 14.2 -3.2
North Carolina 13 46.7% 3 6.1 -3.1
Ohio 16 41.8% 4 6.7 -2.7
Michigan 14 49.4% 5 6.9 -1.9
South Carolina 7 40.2% 1 2.8 -1.8
Indiana 9 42.2% 2 3.8 -1.8
Georgia 14 39.7% 4 5.6 -1.6
Oklahoma 5 28.1% 0 1.4 -1.4
Alabama 7 33.7% 1 2.4 -1.4
Florida 27 45.7% 11 12.3 -1.3
Utah 4 33.4% 0 1.3 -1.3
Kansas 4 31.4% 0 1.3 -1.3
Tennessee 9 35.3% 2 3.2 -1.2
Wisconsin 8 52.1% 3 4.2 -1.2
Missouri 8 39.4% 2 3.2 -1.2
West Virginia 3 33.5% 0 1.0 -1.0
Louisiana 6 31.0% 1 1.9 -0.9
Nebraska 3 28.4% 0 0.9 -0.9
Iowa 4 45.3% 1 1.8 -0.8
Kentucky 6 29.3% 1 1.8 -0.8
Mississippi 4 39.8% 1 1.6 -0.6
Arkansas 4 12.8% 0 0.5 -0.5
Colorado 7 49.5% 3 3.5 -0.5
Maine 2 52.0% 1 1.0 0.0
New Mexico 3 56.0% 2 1.7 0.3
Hawaii 2 78.7% 2 1.6 0.4
Washington 10 55.3% 6 5.5 0.5
New Jersey 12 54.2% 7 6.5 0.5
Rhode Island 2 65.1% 2 1.3 0.7
New York 27 63.5% 18 17.2 0.8
Minnesota 8 51.8% 5 4.1 0.9
New Hampshire 2 51.6% 2 1.0 1.0
Nevada 4 50.5% 3 2.0 1.0
Oregon 5 58.4% 4 2.9 1.1
Illinois 18 54.0% 11 9.7 1.3
Massachusetts 9 83.9% 9 7.5 1.5
Connecticut 5 63.5% 5 3.2 1.8
Maryland 8 63.0% 7 5.0 2.0
TOTAL 353 47.7% 145 166.7 -21.7

In 17 of these 39 states, Republicans won at least one full seat more than expected from their two-party vote share, totaling 30.5 “extra” seats, while in seven states, Democrats won at least one full seat more than expected from their two-party vote share, totaling 9.6 “extra” seats.

So is this clear evidence that gerrymandering cost Democrats 21 or 22 U.S. House seats in 2016 (nearly the 24 they need to win back control)? Or, as suggested earlier, is it simply that parties with SVR>1.00 simply do a better job of winning close elections?

The marginals have vanished. Let’s define a “close” election as one decided by less than 10 percentage points and/or where the winning candidate received less than 55% of the total vote. In the four states where Republicans won at least two full seats more than expected from their two-party vote share (Pennsylvania, Texas, North Carolina, Ohio; total=12.3 seats), there were only four close elections (out of 83); Republicans won three of them for a net of two seats. There were zero close seats in the three states (Massachusetts, Connecticut, Maryland; total=5.3 seats) where Democrats won at least 1.5 more seats than expected from their two-party vote share.

In fact, only 47 U.S. House races (10.8%) met this definition of “close” in 2016. Republicans won 27 of them, for a net of seven seats–only one-third of the total advantage Republicans had in winning U.S. House seats in 2016 using the SVR measure.

In 2016, Democrats won their seats by an average 43.4 percentage points, while Republicans won their seats by an average 34.4 percentage points: the vast majority of U.S. House seats are extremely safe. Even if you exclude the 59 CD (13.6%) where candidates of one major party ran essentially unopposed[3] (31 Democrats, 28 Republicans), the average winning margin was still 35.1 percentage points for Democrats and 26.9 percentage points for Republicans.

And that leads us to a second measure of redistricting “fairness”: extraneous votes (ExV).

To win a U.S. House election, you simply need to receive one vote more than your nearest opponent; all other votes are extraneous. For example, Democrat Carol Shea-Porter defeated Republican Frank Guinta in New Hampshire CD 1 by 162,080 to 157,176 votes. Ms. Shea-Porter only NEEDED 157,177 votes to win, meaning that she received 4,903 “extraneous” votes. By contrast, Democrat Dwight Evans defeated Republican James A. Jones in Pennsylvania CD 2 by 322,514 to 35,131, fully 288,382 votes more than he needed.

This matters because, ever since Governor Gerry and his salamander-shaped state senate district, partisan gerrymandering has meant packing as many of one party’s voters (Party A) into as few (very safe because filled with extraneous voters) legislative districts as possible. The other party’s voters (Party B, in control of redistricting) would then be spread across the remaining (somewhat less safe, with fewer extraneous voters) legislative districts. Evidence for partisan gerrymandering would thus be if one party tends to have more extraneous votes.

And there is such evidence. In 2016, Democrats averaged 112,222 ExV and Republicans averaged 98,582, meaning Democrats averaged 13.8% more ExV than Republicans[4]. Narrowing the analysis only to the 39 states where partisan redistricting is even possible closes the gap: 111,401 to 102,963, with Democrats averaging 8.2% more ExV. Further removing seats with candidate(s) of only one major party reduces the absolute gap to 97,701 to 89,970, with Democrats averaging 8.9% more ExV than Republicans.

Table 2: Extraneous Votes for Winning U.S. House Candidates, and State Seat-Vote Ratios for Eight States, 2016

State Average Democratic ExV 2016 Democratic U.S. House Seats 2016 Average Republican ExV 2016 Republican U.S. House Seats 2016 2016 SVR
Pennsylvania 181,773 5 106,175 13 0.61
Texas 82,805 11 103,112 25 0.78
North Carolina 126,537 3 112,303 10 0.49
Ohio 135,747 4 71,190 12 0.60
Michigan 122,686 5 73,650 9 0.72
South Carolina 107,847 1 82,484 6 0.36
Indiana 135,898 2 94,554 7 0.53
Georgia 165,971 4 143,680 10 0.72
OVERALL 125,430 35 100,805 92 0.60
Minus Texas 144,967 24 99,944 67 0.57

There were eight states in 2016 where Republican U.S. House candidates won at least 1.5 more seats than expected from their two-party vote share (Table 1), for a total of 19.3 seats. Across these eight states, Democrats were also well behind on both measures of redistricting unfairness, averaging 24,625 more ExV and a SVR of 0.60 (without Texas: 45,023 ExV and SVR=0.57). Following the 2010 elections (and thus the 2010 U.S. Census), all but North Carolina had a Republican governor and all had state legislatures controlled by Republicans.

There is thus reasonably compelling evidence that Democrats were on the losing end of partisan redistricting in these eight states that was not nearly countered by controlling the post-2010 redistricting process in such states as Massachusetts, Connecticut and Maryland.

What this means for Democrats—and for democracy. There is nothing Democrats can do until after the 2020 U.S. Census to reverse the post-2010 redistricting. The first step, of course, would be to control governor’s mansions in these eight states in Table 2 following the 2020 midterm elections. This would require Democrats to hold governor’s mansions in North Carolina and Pennsylvania; win open races in Georgia, Michigan and Ohio; and defeat incumbent Republican governors in Indiana, South Carolina and Texas. They also need to chip away at Republican control of these state legislatures. If I were in charge of the Democratic Party, this is where I would focus my energy.

But there are two larger problems that I see, neither of which is good for our democracy because of the ways they impact partisan polarization and the responsiveness of elected  officials.

The first problem stems from a limiting factor on drawing legislative district lines: where each party’s voters live. If Democratic and Republican voters are scattered evenly across states, partisan redistricting is much harder.

The fact, however, is that voters are NOT evenly distributed across states. Democrats primarily self-segregate into large urban areas and college towns, the Atlantic and Pacific Coasts, and majority-minority counties (e.g. the southern “Black Belt,” northern New Mexico, southwestern Texas). And while this self-sorting may allow Democrats to win state races by winning large majorities in a few overwhelmingly Democratic counties, it also makes it easier for Republicans to create a few safe Democratic CDs and state legislative districts in a state while carving up the rest of the state for themselves.

To be fair, many of these CDs are majority-minority as dictated by the Voting Rights Act, but there is a difference between winning 90% of the vote and winning 55% of the vote. To a large extent, Democrats have been complicit in adverse redistricting by allowing for the creation of super-majority-minority CDs.

For example, consider the 2016 election results in Pennsylvania (Figures 3 and 4, taken from here and here):

Figure 3: Pennsylvania Counties won by Democrat Hillary Clinton (Red) and by Republican Donald Trump (Blue) in 2016

Pennsylvania 2016 Clinton-Trump

Figure 4: Pennsylvania Congressional Districts won by Democrat (Blue) and Republican (Red) in 2016

Pennsylvania 2016 CDs

Just six of Pennsylvania’s 67 counties (9.0%) accounted for 57.9% of Hillary Clinton’s vote in that state: five southeastern counties around Philadelphia and Allegheny County (Pittsburgh). In total, she won 11 Pennsylvania counties (16.4%), accounting for 67.1% of her statewide total.

Democrats won only five of 18 Pennsylvania CDs in 2016. Three of them (1,2,13) are clustered in the southeastern corner of the state (average winning margin 81.6 percentage points); the Democrats also won a Pittsburgh-area CD (14) by 48.7 percentage points and a Scranton-area CD (17) by 7.6 percentage points.

Do you see the problem?

Democrats won CDs 1, 2 and 13 in 2016 by an average 239, 757 votes; they are bordered to the north, south and west by CDs 6-8, which Republicans won by an average 52,676 votes. If Democrats control the post-2020 redistricting in Pennsylvania, they theoretically could redraw CD boundaries to shift enough Democratic voters from CDs 1, 12 and 13 into CDs 6-8 to make the latter CDs no worse than swing districts, while keeping the former CDs in Democratic hands, partially winning back the 3.3 “extra” seats Republicans won in 2016 (Table 1).

But nothing would be guaranteed and Democrats could end up violating the rules of compactness and relative population equality. Also, the population of Pennsylvania CD 2 is 61.2% Black; redrawing it could run afoul of the Voting Rights Act.

The other two Democratic seats in Pennsylvania offer little aid. Michael Doyle won his CD 14 by 167,294 votes, while Republicans won the bordering CDs (12, 18) by 84,498 and 293,684 votes, respectively; Tim Murphy was unopposed in the latter race. And Matt Cartwright only won his CD 17 by 22,304 votes; not very many Democratic votes to spare there.

So, Democrats’ best chance to make Pennsylvania’s redistricting more “fair” lies in just three CDs in the southeastern corner of the state because that is where the vast majority of the state’s Democrats are. It would almost be easier for Democrats either to win (back?) voters in the rest of the state or to redistribute themselves across the state, neither of which is necessarily all that easy.

And this is broadly true for Democrats nationwide, who would need to flip a net 24 CDs to win back control of the U.S. House in 2018 (since 1970 the average seat loss for the party holding the White House in its first midterm election [n=7] is 24.1). The path of least resistance is to win back the 27 “close” seats won by Republicans in 2016, while losing no more than three of the 20 “close” seats they won; Republicans won 12 of those 27 seats by more than 10 percentage points.

Talk about threading the needle!

This leads us to the second large problem. Even though Democrats are behind on both the ExV and SVR measures, Republicans also had very high average winning margins and extraneous vote totals. BOTH parties are drawing safer and safer CDs for themselves, a trend Yale political scientist David Mayhew first pointed out in 1974.

This lack of competition makes U.S. House members (and legislators generally) more worried about being challenged in a primary (from the left for Democrats and from the right for Republicans) than they are of losing the “swing” voters. When elected officials are more responsive to the median voter in their own party than they are to the median voter in their district (or state), their incentive to cooperate and compromise with elected officials of the other party diminishes accordingly. This leads to legislative gridlock, meaningless “gotcha” votes and increased political polarization.

If partisan gerrymandering exists, and if it then leads to such perverse incentives, then it should viewed not as threat for one political party, but for our democracy more generally.

Until next time…

[1] I use two-party vote (i.e., the percentage Democratic and percentage Republican of the votes cast only for those two parties) because I will, inter alia, compare percentage of votes won to U.S. House seats held. Every House member is either a Democrat or a Republican, so including third-party votes in calculating these percentages would be misleading.

[2] Hawaii and New Jersey use a “political commission” for redistricting. As I am uncertain how free from partisan pressures these commissions are, I choose to include them from these analyses.

[3] Or faced primarily third-party opposition, including the three Arkansas Republicans whose closest opponent was a Libertarian.

[4] In 34 CDs, only one major party fielded candidates while smaller parties (usually Libertarians) fielded candidates. the extraneous votes is relative to “all other votes combined.” This increased the average ExV for Democrats and Republicans, only marginally affecting the analysis, as shown later in the paragraph.