Is there a ‘Bradley effect’ in U.S. election polling?

With Obama ahead in opinion polls, the ‘Bradley effect’ – where polls overestimate the vote share of black candidates – is coming up for discussion. How large is the Bradley effect? In fact, does it still exist at all? A comprehensive quantitative analysis says “no”.

[ I'm veering off the beaten track here with a brief foray into the world of U.S presidential elections. The information in this post isn't new or obscure, I've just put it together to satisfy by own curiosity. I do have a passive (i.e. non-voting) interest in the U.S. presidential election, but this post is more about opinion polling than about politics. ]

With under two weeks to go to the U.S. presidential election, opinion poll aggregates show Obama with a stable and decisive lead of  7.0 ± 1.1% percentage points ^[1]. The stated error margin is small, but can these opinion polls be trusted? Specifically, are opinion polls reliable when an African American candidate is contesting a U.S. election?

Three high profile elections in the eighties suggested that there was a systematic offset between the results of opinion polls and the final outcome of the election when a black candidate was on the ticket. In these three elections, the Democratic Party candidates were black, and their Republican Party opponents were white:

Mayor Thomas J. Bradley

• Governor of California, 1982:
  Thomas J. Bradley (Dem.) vs. George Deukmejian (Rep.)
  Bradley had a 7% poll lead, lost by 1.2%.
• Governor of Virginia, 1989:
  L. Douglas Wilder (Dem.) vs. Marshall Coleman (Rep.)
  Wilder had a 9% poll lead, won by 0.4% ^[2].
• Mayor of New York, 1989:
  David Dinkins (Dem.) vs. Rudy Giuliani (Rep.)
  Dinkins had a 14% poll lead, won by 2% ^[2].

In these three cases, opinion polls immediately before the election overestimated the black candidate’s vote margin by 8%–12%. This huge polling skew became known as the ‘Bradley effect’ or the ‘Wilder effect’.

How did the polls get it so wrong? The inference was that some voters were lying to the pollsters. To put it less harshly, they were giving what they saw as the socially desired reply, rather than their true voting intention. “Social desirability bias” is a fairly common problem in opinion polls ^[3], it’s not specific to the race issue.

To be clear, the Bradley effect is about voters misleading pollsters, it’s not about voters’ motives when voting:

Denying the existence of the Bradley Effect does not mean denying that some people vote on the basis of race. I have no doubt that some people will vote against Barack Obama because he is black.
[...]
But the Bradley Effect is not an argument about whether people vote based on race. It’s an argument about whether people will lie to pollsters. So long as race-based voters are honest about their intentions, Barack Obama’s position is no worse than it appears to be in the polls.

Nate Silver,
FiveThirtyEight, August 11, 2008 ^[4]

With Obama putatively ahead in the polls for the presidential election, is the Bradley effect relevant today? Kathy Frankovic, Director of Surveys at CBS News, doubts it:

From CBS:  Analyzing the Bradley Effect

2:53

The ‘Bradley effect’ is shrouded in uncertainty. Counterexamples exist where polling results were skewed in the opposite direction, and the effect has disappeared altogether in some recent U.S. elections. The discussion is often a confused jumble of conflicting anecdotes about individual electoral campaigns.

Yet the issue is amenable to empirical analysis; we can move beyond anecdote. Does the effect really exist? If so, how large an offset does it introduce between opinion polls and real election outcomes? In a new study, Daniel J. Hopkins has performed a large-sample quantitative analysis of this question ^[5]. He’s moved the discussion beyond case-by-case anecdotes.

Hopkins’ Paper

Hopkins collected polling data and election outcomes for the 133 Senate and Gubernatorial elections from 1989 to 2006 where a woman or an African American was a candidate of a major party, and for which polling data could be found. While the election returns and turnout data were easily obtainable, polling data were a little trickier to get, and he obtained them by a database search of newspaper archives.

He looked for systematic polling offsets for both female candidates and black candidates. He found no polling skew over any time period for female candidates (there is no “Whitman effect” for female candidates) but the situation is more complicated for black candidates. Over the 1989–2006 time frame, there were 18 elections where a black candidate ran against a non-black candidate, and Hopkins obtained 47 opinion poll results for those elections. Figure 1 shows the difference between poll share and vote share plotted against time for those 47 points. The graph is reproduced from Hopkins’ paper.

Figure 1.  The gap between a black candidate’s share in a poll and subsequent share of the vote, for the 18 Senate and Gubernatorial elections in the period 1989–2006 when a black candidate ran against a non-black candidate.

Notes: The graph is reproduced from Hopkins, Ref [5]. The data are for 18 Senate and Gubernatorial elections between 1989 and 2006. The 47 data points correspond to the 47 available poll results. Only polls taken within 1 month of the election are included. The “gap” is the candidate’s poll share minus vote share. A positive gap means the poll share exceeded the eventual vote share. The blue line is a locally weighted polynomial fit called a LOESS model. The polynomial is fit using weighted least squares, giving more weight to points near the point whose response is being estimated and less weight to points further away.

The study’s core conclusion is that the Bradley (or ‘Wilder’) effect existed up to the mid-1990s, but has since declined:

How to explain the absence of the Wilder effect in these data when elections in the late 1980s made its existence so indisputable? Figure 1, which plots the polling-performance gap by year alongside loess smoothing lines, provides a preliminary answer. In the first several years, all of the observed gaps are positive, meaning that black candidates consistently polled better than they performed. The Wilder effect was at work. Yet in 1994, we see black candidates such as Washington’s Ron Sims performing worse than their polls for the first time. And as of the late 1990s, the Wilder effect disappears entirely. Even Tennessee’s 2006 Democratic nominee for Senate, Harold Ford Jr., experienced no Wilder effect after a negative television advertisement targeting him cued anxieties about inter-racial sex.
[...]
Before 1996, the median gap for black candidates was 3.1 percentage points, while for subsequent years it was −0.3 percentage points. Even for this relatively small sample, the p-value on the t-test that these two distributions have the same mean is 0.01. That rejects the possibility that the differences across years stem from chance alone.

Daniel J. Hopkins ^[5]

Hopkins checked the Bradley effect’s post-1996 disappearance for omitted variable bias, and found that the result was robust to a range of potentially omitted variables. One important additional factor did emerge though, that of ‘front-runner decline’:

Exploratory analyses uncovered an aspect of these data that proves critical in isolating the Wilder effect. The main candidates who suffered from the Wilder effect were all front-runners, including Bradley, Dinkins, and Wilder. If front-runners commonly experience a decline from their surveys to their performance at the ballot box, the racial aspect of the Wilder effect might be overstated. To check this possibility, the analysis first turned to female candidates, a group that does not suffer from any polling-performance gap. Among candidates polling above 50%, the average polling-performance gap was 1.9 percentage points, meaning that they tended to do worse at the polls than surveys would predict. For female candidates who were behind in the polls, they performed an average of 2.5 percentage points better on election day. The polling-performance gap depends on a candidate’s initial level of support.

Daniel J. Hopkins ^[5]

Opinion polls overestimate the electoral support of front-runners, and underestimate the support of lower-placed candidates, possibly because respondents simply give the name they’re most familiar with, the one that first comes to mind. Hopkins examined the front-runner effect numerically and concluded that some, but not all, of the Bradley effect can be attributed to ‘front-runner decline’.

Why did the Bradley effect disappear after 1996? Hopkins offered some speculative suggestions, but this kind of study cannot provide an answer to that question:

During a racially charged campaign, a candidate’s race will be foremost in voters’ minds. According to this racialization hypothesis, we might expect abrupt changes in the Wilder effect as the information environment shifts. The focus in this paper is on the national information environment, as it has observable implications for many elections, although campaigns certainly can racialize local and statewide streams of information as well. In the period under study, the prominence of racialized issues such as crime and welfare declined markedly at the national level in the late 1990s and early 2000s. [...] Both of these issues are closely connected to racial attitudes, yielding a prediction: for black candidates, the Wilder effect should have declined during this period. In the early 1990s, white voters would have been primed by national politics to connect blacks and black candidates to issues such as crime and welfare. By the turn of the 21st century, those associations were less salient nationally.

Daniel J. Hopkins ^[5]

An alternative possibility is that pollsters have worked out how to remove this systematic polling error at their end, but Hopkins hasn’t examined this side of things in his paper.

Hopkins’ paper has been extensively blogged. See Jon Cohen ^[6], Marc Ambinder ^[7], Sam Wang ^[8] and Andy Eggers ^[9] for more information.

Hopkins’ analysis looked at Senate and Gubernatorial elections only. Might voter behaviour change in a presidential election? We’ll see next week. I’ve put up this post ahead of the election to pin down the current state-of-play, I’ll come back with a short update when it’s all over.

And One Last Point…

The ‘Bradley effect’ got its name from the Bradley-Deukmejian election, but was that election even an example of the Bradley effect? Blair Levin was on Tom Bradley’s campaign team in 1982, and had an inside view of that election. In Levin’s opinion, Bradley’s skewed poll data weren’t caused by the Bradley effect at all, but because pollsters had overlooked a surge in absentee ballots. He’s unhappy that Bradley’s name is even associated with the term ^[10]:

On election night in 1982, with 3,000 supporters celebrating prematurely at a downtown hotel, I was upstairs reviewing early results that suggested Bradley would probably lose.

But he wasn’t losing because of race. He was losing because an unpopular gun control initiative and an aggressive Republican absentee ballot program generated hundreds of thousands of Republican votes no pollster anticipated, giving Mr. Deukmejian a narrow victory.
[...]
As we’re on the subject, we should free Tom Bradley’s name from an association he would have abhorred. After all, he practiced the sort of politics whose goal was to bring people together, not to play up their differences.

Blair Levin,
The New York Times, October 19, 2008 ^[10]

Levin also makes this point on the BBC’s Today programme on Oct. 25, 2008:   audio.

Tom Bradley was Los Angeles’ first African-American mayor. Here’s a glimpse of Bradley beyond the ‘Bradley effect’:

The New Los Angeles

2:49

Update

Here’s the post-election update: “So, was there a ‘Bradley effect’? (Answer: No)”.

References

1. In which I write of paint continuing to dry, Sam Wang, Princeton Election Consortium blog, October 22, 2008
2. Can You Trust What Polls Say about Obama’s Electoral Prospects?  Scott Keeter and Nilanthi Samaranayake, Pew Research Center, February 7, 2007
3. Methods of coping with social desirability bias: A review, Anton J. Nederhof, European Journal of Social Psychology 15, 263–280, 2006
4. The Persistent Myth of the Bradley Effect, Nate Silver, FiveThirtyEight, August 11, 2008
5. No More Wilder Effect, Never a Whitman Effect:  When and Why Polls Mislead about Black and Female Candidates, Daniel J. Hopkins, Department of Government, Harvard University, August 4, 2008  (preprint)  (WebCite cache)
6. New Research on the “Wilder effect”, Jon Cohen, Behind the Numbers blog, The Washington Post, July 29, 2008
7. Has The Wilder Effect Disappeared?  Marc Ambinder blog, The Atlantic, September 19, 2008
8. The disappearing Bradley effect, Sam Wang, Princeton Election Consortium blog, September 27, 2008
9. Timely research: Hopkins on the Wilder Effect, Andy Eggers, Social Science Statistics Blog, October 1, 2008
10. What Bradley Effect?  Blair Levin, The New York Times, October 19, 2008