The political polling world has been roiling this week over a new survey showing that President Obama's approval rating has sunk to 50 percent, with his disapproval numbers rising to that same level.
But it's not just the numbers that propelled pollsters into a serious chat room snit — though they represent a marked departure from other recent polls that consistently show the president with approval ratings hovering around 60 percent.
It's the Internet survey methods that pollster John Zogby used to gather the data that have research-centric Web sites and forums brimming with agita. Zogby's technique, critics say, turns the traditional polling practice of random survey on its head by drawing poll participants from visitors to his site who volunteer to be a member of his "online panel."
Methods Ahead Of the Curve, Or Just Off?
Zogby says he also recruits panel members during traditional phone call surveys and from purchased e-mail lists. And the pollster stoutly defends his method as ahead of the curve in its use of the Internet.
However, Peter Miller, president-elect of the American Association for Public Opinion Research, says that other surveyors find "legitimate reason to be worried about the Zogby results."
There have been questions about Zogby's political polls in the past — especially after he incorrectly predicted key states would vote for Democratic presidential candidate John Kerry in 2004. Zogby himself offers up the information that influential news organizations including The Associated Press and ABC won't use his data.
But the spirited discussion around the Zogby method reflects a much broader issue faced by the public research community: Its time-honored phone-people-at-home survey method has been pushed to the brink of obsolescence by the wholesale transformation of the way people communicate.
Dealing With The Digital Revolution
Every pollster worth his or her salt is desperately searching for a new model that will continue to feed America's appetite for survey results while remaining consistently trustworthy — without being able to rely on old-fashioned land-line phone calls.
Like it or not, that new equation will almost necessarily involve using Internet queries, as well as text messaging, e-mail and a much broader representation of cell phone users.
"We do have a legitimate problem with the fact that response rates to telephone surveys are going down and we don't have something that costs roughly the same that we can use to replace the telephone," Miller says.
Zogby isn't the only pollster who uses the much-cheaper Internet method to gather survey data. So why is he such a lightning rod?
The science of polling is in the sample — a random sample, says Iowa-based pollster J. Ann Selzer.
"That means that every person who is qualified to participate has an equal chance of being selected to participate in the poll," she says.
But Is It Scientific?
To create that sample, pollsters use random-digit phone numbers so each phone — listed or unlisted — has an equal chance of being called. They also use voter registration lists and pull names randomly.
"Of course," Selzer says, "in this day and age, there is no single method that gives 100 percent of qualified respondents an equal chance to participate. Still, the comparison of a random-digit phone sample and an online panel is apples and Ugli fruit."
Zogby's online panel members, who he says number about 1.1 million, have all agreed to be polled. That means they haven't been chosen randomly. It's that self-selecting element that troubles experts like Mark Blumenthal of www.pollster.com.
"His is only a random sample of those who agreed to be part of his sample," Blumenthal says. He noted that a review of pollster results from past election years showed that Zogby's "particular Internet method is less reliable" than other polling methods.
But Zogby, who says he is used to attacks on his polling, argues that people who agree to a telephone survey have also made a conscious decision to participate.
"Who in the hell are the people who answer the telephone? They're self-selected, too," Zogby says. "And pollsters are only getting a 24 percent response rate from those calls." In the golden days of telephone polling, he said, that response rate exceeded 60 percent.
His method, Zogby says, "is not perfect." Consistent Zogby critic Nate Silver on the polling site www.fivethirtyeight.com says that Zogby missed badly in calling states in the last presidential election.
Silver says that in Internet presidential preference polls Zogby conducted for 11 states last October, he missed three of 11 states and overall was off on the final margin of error by an average of 5.4 percentage points — double the average margin-of-error miss by all the other polls in the same states.
"All told," Silver noted, "Zogby's Internet polls have been off by an average of 7.6 points." That means his polls, like his recent Obama approval survey, are considered extreme outliers in the world of political research.
And so it goes.
A 'Huge' Challenge
Zogby says he is a pioneer, and perhaps he is. But it's an emerging survey world, one still fraught with problems.
"The online polling world is going to be a big part of the future of polling, but it still has a ways to go," says pollster Scott Rasmussen. "I think there's great potential, but it's in its infancy."
Blumenthal says that the survey research profession faces a "huge, difficult and interesting challenge."
"We need to experiment — to let a thousand flowers bloom," he said.
And if they can't come up with a recipe for the perfect polling stew — the one that will allow a taste to represent the whole?
There's always the possibility of returning to the old-fashioned way of doing things, says Miller. Forget the Internet, text messaging or e-mail: Just mail out surveys they way researchers used to, or get outside and knock on some doors. Not cheap, but in the diffuse world of communications, perhaps the only path to accuracy.
"Unfortunately for us," Miller says, "we live in interesting times."
Copyright 2022 NPR. To see more, visit https://www.npr.org. 9(MDAzMjM2NDYzMDEyMzc1Njk5NjAxNzY3OQ001))
