Exit poll error? It’s the Shy Tory Factor, not the Bradley Effect

While early exit polls showed Sen. Barack Obama leading in the Pennsylvania Democratic presidential primary, and broadcast and cable news networks consequently delayed calling a winner in the election, when the actual ballots were counted Sen. Hillary Clinton had carried the Keystone State by some 9 percentage points.

Some pundits quickly suggested that the gap between the exit polls and the final tally reflected the Bradley Effect—white voters telling pollsters they had voted for the black candidate, when in fact they had not (an effect first identified in California’s 1982 gubernatorial election lost narrowly by Tom Bradley, an African-American—hence the name.) It wasn’t the first time in Campaign 2008 that Obama’s strong unweighted exit poll numbers did not translate into actual votes—the Illinois Senator had “underperformed” in New Hampshire and in several Super Tuesday states, according to a compilation of early exit polls by Brendan Loy. Loy further noted that: “… Obama generally does 7-8 points worse in the actual results than he did in the leaked, unweighted exit polls.”

But it’s unlikely that the color of the candidates caused the exit poll problems. Instead, it appears that the Shy Tory Factor influenced the exit polls in Pennsylvania, a global phenomenon that has surfaced in numerous past elections where race wasn’t a consideration. The Shy Tory Factor is when conservative voters provide misleading answers to pollsters or refuse to participate in exit polls (where it is called “non-response bias” by pollsters). It has been seen in elections in England, France, Italy, Australia and the U.S. More conservative candidates perform better at the ballot box than they do in pre-vote polls and exit polling. (Reuters, for example, noted that “exit polls have not always proved reliable in Italy” in reporting on the recent election of conservative candidate Silvio Berlusconi.)

What’s behind it? The prevailing theory is that these Shy Tory, or Shy Conservative, voters opt out of polling, or offer misleading answers, because they don’t view the elite media, who sponsor the opinion and exit polls, as truly neutral. They realize that their candidate (Berlusconi, Margaret Thatcher, Ronald Reagan, George W. Bush, John Howard) is not the choice of liberal reporters or mainstream commentators and consequently they are more reluctant to share their preference with intrusive pollsters.

The practical effect of the Shy Tory Factor is to skew poll results. Take Pennsylvania. While Clinton supporters may not be “Tories” in ideological terms, they are older, less-educated, and more likely to resent the media anointment of Obama. Their motives for refusing match the Shy Tory model. If these Clinton voters shied away from exit polls, it means backers of the other candidate (Obama) were oversampled. The Clinton-Obama race had further complications. Refusal rates for exit polls are historically greater among older voters to start with. Most exit poll takers are young (students, etc.), and it would not be surprising if—despite being trained to avoid interviewer-caused selection bias—these temporary workers gravitated to polling younger voters, who have continued to favor Obama.

Vote fraud?

There is, of course, a third possible explanation for the disconnect between the exit polls and the tabulated vote—that of election fraud. After the New Hampshire primary, some on the Left suggested that Sen. Clinton’s victory involved rigged voting machines, and others (such as posters on The Brad Blog and TruthDig ) have questioned the validity of the Pennsylvania primary as well.

A common misunderstanding about the accuracy of exit polls has contributed to these conspiracy theories. (“Mystery Pollster” Mark Blumenthal has researched exit polling’s historical inaccuracy.) They simply aren’t a valid way to audit elections. For starters, exit polls carry a margin of error—supposedly about 3 percent in national elections, when all else goes well, and higher in primaries. And like all polling, exit polls rely on a representative sample of voters that is projected to all voters (which is what the “weighting” process is all about). As can be seen with the Shy Tory Factor, when given voters won’t participate, it skews the sample. (See this interview with Joe Lenski of Edison Media Research for a frank assessment of the problems with exit poll samples). The exit poll refusal rate has been growing in the U.S. It was an average 35% nationwide in the 2004 presidential election and is higher for older voters and in more conservative areas of the country.

These flaws were ignored by Democratic activists and bloggers when, as evidence of fraud, they pointed to those pivotal states where exit polls had projected Sen. John Kerry as the winner but where President Bush triumphed when the actual votes were tallied. In explaining the discrepancy, Edison/Mitofsky Research (the firm that conducted the 2004 exit polls) concluded that Republican voters had refused to participate in exit polling in greater numbers than Democrats, leading to an overestimation of the Kerry vote totals. Further voter sampling problems surfaced in the 2006 Congressional election exit polls.

None of this will, however, convince the conspiracy buffs who believe that the Clinton machine—borrowing alleged Republican tactics—manipulated the primary voting process in states like New Hampshire and Pennsylvania by a conspiracy targeting electronic voting machines. Such a vote fraud enterprise in practice would require the complicity of hundreds of election officials, computer technicians, etc., spread across numerous precincts, and would also demand a breathtaking level of coordination and planning. And everyone involved would be committing numerous felonies as well. But if you believed the Republicans were capable of such crimes in 2004, it’s not as hard to believe that the Clinton campaign would engage in vote fraud as well.

A media creation

There are some ironies in these exit poll problems. Exit polls, after all, are a media creation. They allow network anchors and political commentators to pontificate about voter preferences and beliefs. They allegedly tell us how given groups (whites, blacks, Hispanics, liberals, conservatives, Catholics, Jews) voted, and why they voted the way the way they did (contributing, one could argue, to the public practice of identity politics). If media cheerleading for Obama has increased exit poll refusal rates among Clinton voters, then the lack of balance in the coverage of the Democratic race in 2008 has contributed to the margin of error in these surveys.

The Edison/Mitofsky Research folks don’t like to talk about refusal rates, because they know it raises questions about the validity of their exit polls. A strong argument can be made based on the 2004, 2006, and 2008 results that the accuracy of exit polls has been so compromised that they should be abandoned as an analytical tool in political news coverage. And other than a few media executives and polling firms, who would be sorry to see them go?

Copyright © 2008 Jefferson Flanders

All rights reserved

Add to Technorati Favorites!

Campaign 2008’s troubled opinion polls

Can’t say I envy political pollsters during this topsy-turvy Campaign 2008. Who likes to be embarrassed on a national (if not global) stage by getting it wrong?

All of the major pre-vote polls predicted a Barack Obama victory in Tuesday’s Democratic New Hampshire presidential primary. All were wrong, as Sen. Hillary Clinton confounded conventional political wisdom and predictions of pundits and campaign insiders and won, 39% to 37%, over Sen. Obama.

One explanation offered by many in the “political industrial complex”, such as Frank Newport of Gallup and long-time political researcher Peter Hart, is that pollsters stopped surveying too early, and missed a late shift by women voters and independents to Clinton. The historical precedent for this last-minute decision scenario: the presidential election of 1948 (“Dewey Defeats Truman”), when polling shut down prematurely and missed Harry Truman’s late surge.

The truth is that we don’t know for sure what happened. While most of the polls correctly predicted the outcome of the Republican primary (Sen. John McCain over former Gov. Mitt Romney), the predicted margins of victory for Sen. McCain varied widely, and one poll (Suffolk-WHDH) picked Romney as the GOP winner.

The flawed New Hampshire pre-vote polls suggest that Campaign 2008 may challenge the ability of opinion polls to predict trends or electoral outcomes with any certainty.

Here are five factors that will spell trouble in the months ahead for anyone looking to rely on political opinion polls.

1. Traditional turnout models may not be able to predict who will show up to vote in a non-traditional year when the candidacies of a woman and an African-American are drawing new “wildcard” voters to the polls (a point first made by Frank Luntz, pollster for Fox News). Gauging the intentions of prospective voters is very difficult; in the past the methodology has relied on a series of predictors (such as homeownership and voting in the last election) that may no longer be as valid.

2. Round-the-clock news coverage has made for a more volatile political environment, as prospective voters are quickly made aware of any campaign developments, positive or negative. The 24X7 news cycle of the Internet and the cable news networks means that voters can be exposed to potential “decisive moments,” like Sen. Clinton’s emotional Portsmouth cafe cry, in the last days and hours of a race. If the 2008 primaries feature more last-minute voter decisions, it will be hard for polls to reflect his new reality.

3. Perceived media cheer-leading for the Obama and Clinton candidacies because of their historic nature (the first credible female and African-American presidential candidates) may make some voters hesitant to reveal their true intentions. This is a variation of the Shy Tory or Shy Republican Factor, where voters won’t admit to pollsters that they plan to vote for conservative candidates. (Right-of-center politicians, such as Margaret Thatcher, Ronald Reagan, John Howard and Silvio Berlusconi, have often performed better in the election than in pre-vote polls.)

4. The growing Do Not Call ethos in many American households—which has led to call screening, unlisted numbers, and a general reluctance to answer phone surveys—makes it harder to assemble a representative sample. (While federal law allows political and survey calling, it doesn’t mean such intrusion are welcomed). And reaching those prospective voters who rely solely on cell phones (often younger voters), an estimated 15% of Americans, is another emerging challenge.

5. A growing It’s None of Your Business backlash may be in play. Response rates for political polls have been dropping. Telephone response rates have dropped from about 70% in the 1970s to 30% today (although pollsters insist survey quality hasn’t declined). Consider the refusal rate for exit polls: it’s much greater in the United States than in other countries (I’ve seen non-participation numbers ranging from 10% to 40% of voters, depending on the election). In the 2004 Presidential election, Republican voters refused to participate in exit polling in greater numbers than Democrats; most likely another manifestation of the Shy Tory effect, as some GOP voters may have regarded the exit polls as an extension of the “liberal media.” In fact, the predictive shakiness of the 2004 and 2006 national exit polls has been linked to problems with voter sampling.

What these factors mean for Campaign 2008: the polls, whether pre-election or exit, should be treated with great caution by citizens, pundits, journalists, and political observers. The Democratic New Hampshire primary results should serve as a warning to anyone relying on what is more muddled art than precise science.


Copyright © 2008 Jefferson Flanders
All rights reserved