Evaluating Public Opinion Polls

 

  1. Polls may look convincing yet be completely worthless
    1. Public opinion is subjective and can change rapidly.
    2. Consequently, polls sometimes produce conflicting or meaningless results, even when they are carefully written and presented by professional interviewers to scientifically chosen samples.
    3. Pollsters can also set up surveys that deliberately shade the truth.
  2. Four simple questions can separate the good polls from the trash.
    1. Did they ask leading or biasing questions?
      1. A leading question is one that leads the respondent to choose one response over another by its wording.
        1. Leading questions are actually statements disguised as questions, and make respondents feel that only one response is legitimate.
        2. For example: "Don't you agree that the look and feel of user interfaces should not fall under copyright protection?"
      2. Prestige bias occurs when a statement, position, or response is associated with a prestigious person or group. Respondents may answer on the basis of their feelings toward the person of group rather than addressing the issue.
        1. For example, a survey might state, "According to recent Gallup polls, 80 percent of the people support granting china Most Favored Nation Status. What is your opinion on this issue?"
        2. This way of stating the position of a professional organization might easily sway people.
    2. Did they ask the Right People?
      1. Response bias
        1. In 1936, the editors of Literary Digest conducted a Presidential preference poll of more than 2 million Americans.
          1. The poll predicted that the Republican candidate, Alf Landon, would defeat Franklin Roosevelt.
          2. Landon's loss made the Digest history's most famous victim of sample bias.
        2. The Digest mailed more than 10 million ballots to households listed in telephone books and automobile registration records.
          1. Might create a relatively representative sample today
          2. In 1936, it substantially biased the sample toward those affluent enough to own cars and phones.
        3. George Gallup and Elmo Roper used samples of a few thousand people to predict a Roosevelt win. Gallup and Roper carefully chose their samples to reflect a demographic cross-section of Americans
        4. Some journalists and businesses today still make the mistakes the Literary Digest made 60 years ago.
        5. The temptation to take a biased poll is great if you have a tight deadline and a small budget, as many news organizations do
        6. Also, often produces "surprising" results, stimulating interest.
      2. Non-response bias: Even when you start out with a representative sample, you could end up with a biased one.
        1. Readers of women's magazines are frequently asked to fill out surveys on weighty subjects like crime and sexual behavior
          1. Ignore the opinions of nonreaders
          2. Biased toward readers who take the trouble to fill out and return the questionnaire, usually at their own expense.
        2. Television news and entertainment shows post toll-free or even toll numbers that viewers can call to "vote" on an issue
          1. These samples are biased in the same way
          2. They are also prone to "ballot-stuffing" by enthusiasts. In other words, viewers who call 12 times get 12 votes.
      3. Poll results based on "convenience" samples can be wildly misleading, even if the sample sizes are huge.
        1. A call-in poll conducted by a television network in 1983 asked: "Should the United Nations continue to be based in the United States?"
        2. About 185,000 calls were received.
        3. Two-thirds said that the U.N. should move.
        4. At the same time, the network conducted a random-sample poll of 1,000 people, and only 28 percent said the U.N. should move.
      4. Systematic "intentional" biasing
        1. Conflicts make news. When journalists are trying to liven up a boring political story, they need angry, well-informed citizens
          1. This is one reason why older men may be quoted more often than other groups.
            1. Those aged 50 and older are more likely than younger adults to follow news stories "very closely," and men are more likely than women to follow stories about war, business, sports, and politics.
            2. Two-thirds of regular listeners to political talk radio programs are men, according to a 1996 poll taken by Roper Starch Worldwide for the Media Studies Center in New York City.
            3. Republicans outnumber Democrats three to one in the talk-radio audience, and 89 percent of listeners are white, compared with the national average for voters of 83 percent
            4. Three in five regular listeners to political talk radio perceive a liberal bias in the mainstream media, compared with one in five nonlisteners.
      5. Do not be confuse this with normal sampling error
        1. For example, the CNN/USA Today/Gallup Poll of January 5-7, 1996, showed that the proportion of Americans who approved of President Clinton's performance had dropped to 42 percent, from 51 percent on December 15-18, 1995.
        2. Polls conducted that same week by Washington Post/ABC and New York Times/CBS showed that his approval rating was 53 percent and 50 percent, respectively. This made for a few wild days at the White House, until the next Gallup survey showed a sudden rebound.
        3. Reputable surveys report a margin of error-usually of 3 or 4 percentage points-at a particular confidence level-typically 95 percent
          1. This means that 5 percent of the time, or 1 time in 20, the poll's results will not be reliable.
          2. The other 95 percent of the time, it is within 3 or 4 percentage points of the "truth." This sort of inevitable statistical problem explains the blip in the January Gallup poll.
    3. Which questions came first?
      1. The order in which questions are asked can have a big effect on the results.
        1. In late May 1996, the CNN/USA Today/Gallup poll reported that 55 percent of Americans believe taxes can be cut and the federal deficit reduced at the same time, compared with 39 percent who do not believe this.
        2. The same week, New York Times/CBS reported a dead heat of 46 percent who believe and 46 percent who do not.
          1. This variance was way beyond the margin of error.
          2. The questions were almost identical in their wording
          3. But the order of questions in the Gallup poll may have biased the results
            1. In the CBS poll, questions before the tax cut/deficit question were not related to the subject
            2. Gallup first asked respondents if they favor a tax cut. Then it asked those who did if they would still favor it even if it meant no reduction in the deficit. Then it asked all respondents if they believed both could be done at the same time. By that point, "some of Gallup's interviewees may have felt invested in the idea of a tax cut
      2. Most people want to appear consistent to others and to be consistent in their own minds.
        1. When a pollster asks a series of related questions, this desire can lead people to take positions they might not have taken if they were asked only one question.
    4. What was the question?
      1. Question wording is extremely subtle.
        1. In the hours after President Clinton's November 27, 1995, speech announcing that 20,000 U.S. troops would be sent to Bosnia as part of a NATO peace-keeping mission, three major news organizations took reaction polls. CNN/USA
          1. Today/Gallup found that 46 percent of Americans favored Clinton's plan, while 40 percent were opposed.
          2. CBS found that only 33 percent were in favor, and 58 percent were opposed
          3. ABC said that 39 percent were in favor, and 57 percent were opposed.
          4. The CNN poll was probably more in favor because it did not mention that the U.S. was sending 20,000 troops
          5. CBS and ABC gave respondents the chance to react to that substantial number, which drove down their approval.
          6. In addition, CBS described the troops' mission as "enforcing the peace agreement," while ABC and CNN described the troops as part of "an international peace-keeping force."
          7. CBS's harsher wording may have contributed to its respondents' harsher judgment of the Clinton decision.
      2. Sometimes words are problematic because they are too vague
        1. In April 1996, the Pew Research Center asked which Presidential candidate was best described by the phrase: "shares my values."
        2. By this measure, Clinton beat Dole by 47 percent to 37 percent
        3. But when CBS and the New York Times asked whether each candidate "shares the moral values most Americans try to live by," 70 percent said that Dole did, but only 59 percent said so of Clinton.