SOC 240-01/51 (Kuechler)
Fall 2000
last update: Oct 12, 2000 -- 9 am
 

Case study on the effect of sampling method: Presidential Debate Winner

 
If you received a paper copy of (the first page of) this handout, check out the online version (on the course web page) to follow the embedded links and to view the supporting materials. The direct URL for this document is:
http://maxweber.hunter.cuny.edu/pub/eres/SOC240.00_KUECHLER/debate-polls.htm

All seven surveys were conducted immediately after the conclusion of the first Presidential debate between George W. Bush and Al Gore on Tuesday, October 3, 2000, in Boston. The question was who won the debate. Given the simplicity of the question, the effect of specific question wording (usually a very important aspect to be considered) is negligible. Hence, the differences in results are solely due to sampling method (determining the target sample) and "field administration" (realization of an actual sample). These seven surveys (polls) can be divided into two groups

Under the assumption that the actual samples are truly random samples, a statistical sampling error (often -- especially in the media -- referred to as the "margin of error") can be determined. This assumption, however, is problematic; especially when the actual sample is less than half or even less than a third of the target sample -- in other words: when the "response rate" is relatively low. Unfortunately, few survey organizations provide information about the difference between target and actual sample.
Using the standard statistical formula (which further assumes that a simple random sample was drawn), the sampling error is about 4 percentage points (slightly less for the larger CBS poll). However, this "sampling error" represents only one part of the "total survey error" -- for which there is no easy formula and which is extremely difficult, if not impossible to determine in numerical terms. The term "margin of error" is somewhat misleading, as this figure does not include other errors likely to occur in a survey or poll.

With this caveat: A "margin of error" (better: sampling error) of 4 percentage points means that the "true" percentage in the whole population (here: among all registered voters of which there some 150 Mill.) is plus/minus 4 percentage points starting with the sample percentage. E.g., based on the ABCNews poll, we could say that we are 95% confident that 42 +/- 4 percent -- or between 38 and 46 percent -- thought that Gore won the debate. (95% is a commonly used level, a sheer convention; similar mathematical calculations can be performed at any given "confidence level".)

While all four of the serious polls have Gore ahead, the difference between Gore and Bush ranges from +3 (ABC) to +14 (CBS). Taking the margin of error into account, and based on the ABC and CNN polls, we cannot be 95% confident that Gore is ahead of Bush in the population of all registered voters. (In the media, this situation is often labeled as a "dead heat" -- a very unfortunate and rather misleading term.) But based on the CBS and MSNBC polls, we can be 95% confident that Gore is truly ahead of Bush. So, the four polls lead to different conclusions -- and there is no way to tell which of them is closest to the "true" situation, the percentages among all 150 Mill. registered voters. It is theoretically possible, but highly unlikely, that all four are off the mark and that among all registered voters Bush is ahead -- as the junk polls indicate. An article in the Washington Post ("A quick take on the quick polls") provides additional background on the four serious polls.

Note how much the results of the Web "junk polls" differ from the results obtained with much more rigorous sampling. Of course, methods textbooks use the term "availability" or "convenience" sampling (for which there are also legitimate uses). I use the term "junk poll" to indicate the use of convenience sampling without any justifying scholarly reason. And, in general, junk polls are not restricted to the web; PIPs (phone-in polls) are another genre of junk polls. But I could not find one for this particular case study.
 
Instant Polls after Oct 3 Bush-Gore Debate Sample size Sampling/admin method Percentage 
Gore winner		 Percentage 
Bush winner
ABC News
(TNS Intersearch) 		 491 registered voters Telephone -- RDD  42 39
CBS News
(Knowledge Networks)		 812 registered voters Web -- random 56 42
CNN-USA Today (Gallup) 435 registered voters Telephone -- RDD 48 41
MSNBC-Reuters (Zogby) 536 registered voters Telephone -- RDD 46 36
CNN 40,882 page visitors
61,170 page visitors
61,197 page visitor		 Web -- junk (~8:30am)
(10:50pm)
(final)		 43
44
44		 57
56
56
vote.com 34,568 page visitors
106,411 page visitors
120,236 page visitors		 Web -- junk (~8am)
(10:44pm)
(10/6, 8:19am)
(still open; result page)		 12
29
27		 88
71
73
Harris/Excite 20,291 page visitors
124,268 page visitors
132,280 page visitors		 Web -- junk (~8am)
(10:35pm)
(final)		 54
39
38		 36
46
46
Notes:
1. Results for junk polls are snap shots at specific times, first taken on Oct 4, 2000.
2. The sample for the CBS poll was constructed in several stages. First, a conventional RDD sample; people willing to participate in the study were given WebTV so that they could participate at the designated time via the Web. Check Knowledge Network web site for additional details.
3. Percentages do not add to 100 as there are additional response categories: "neither" and/or "don't know"
4. Links to the instant polls may become invalid as news organizations often display documents for a limited period of time only.
5. The commercial survey organizations actually conducting the polls are listed in parentheses after the sponsoring news organization.

What follows below are "screen shots" taken mostly on the morning of Oct 4, 2000, to document some of the figures shown in the table and to provide illustrations of how web junk poll are conducted. But there is also some information about other serious web based surveys.
 

CNN -- Quick Vote

Typically, the "Quick Vote" box can be found in the middle of the rather long and busy start page of the CNN site. You may need to do some patient scrolling to locate it.


 

Dick Morris' vote.com

A former advisor of President Clinton, Dick Morris is now an independent political consultant. Supposedly this is a non-partisan site aimed at helping to make the opinion of ordinary citizens heard. To participate in the votes, you need to register and supply an e-mail address (which, of course, you can make up). Your vote is also sent via e-mail to the candidates involved (here: Gore and Bush).

I have experienced problems with displaying the results page when using Netscape (4.75), MS Internet Explorer (5.5) seems to work better with this site.



 
 
Vote.com sends how I voted directly to both candidates .....
 

Date: Wed, 4 Oct 2000 07:49:23 -0400 (EDT)
Message-Id: <200010041149.HAA26691@votecom3.cam-colo.bbnplanet.com>
To: kathman@bellatlantic.net
Subject: Your vote - Who Won the October 3, 2000 Presidential Debate?
Sender: "Vote.com" <voting@vote.com>
From: "Vote.com" <voting@vote.com>

Dear Vote.com voter: 

Your vote has been mailed on behalf of kathman@bellatlantic.net to:

Gov. George W. Bush (mail@georgewbush.com) presidential candidate, Bush for President, Inc.
Al Gore (vice.president@whitehouse.gov) Vice President, The White House

The email reads:

On the question Who Won the October 3, 2000 Presidental Debate?, I voted (erased to protect privacy)

Please consider my vote when making your decision on the issue.

Signed,

kathman@bellatlantic.net

If you would prefer not to receive this type of mailing, please click the link below to unsubscribe:
http://www.vote.com/confirm/subscription.phtml?u=59584e7.......

Harris/Excite

Like many other web sites of news organizations or "portals", you can customize the Excite starting page ("Welcome Manfred"; see below), but this is not necessary to get to the poll. I did not take a screen shot right away, when I cast my vote. But I tried to return, I was told that my vote had already been counted (see below). However, if you disable "cookies" for your browser, you can vote as often as you like.

Note that Harris also does serious web based surveys. Harris Interactive claims to be the "Global Leader in Internet-based Market Research". They have assembled a pool of over 7 Million respondents connected to the Internet and supposedly willing to participate in Harris online surveys. As home Internet users are distinctly different form the general population (in particular, they are more affluent and have higher formal education), stratified and weighted random samples are drawn from this pool to approximate a random sample of the general population. In contrast to Knowledge Networks (CBS Poll above), pool members are not given WebTV or any other assistance in order to connect to the Internet. Both methodologies are still in an experimental stage and there are different views among survey research professionals about the current level of achievement and the prospects for the future.

Personally, I strongly believe that telephone surveys will be replaced by web-based surveys as the main mode of administration, but only the future will tell for sure. However, keep in mind that not all web polls are junk polls.




 
 
 
Excerpt from ABCNews web site at
http://abcnews.go.com/sections/politics/DailyNews/DEBATE001004.html

High Stakes 
              The latest ABCNEWS/Washington Post poll shows Gore
              leading Bush 48 percent to 46 percent — a virtual tie given
              the survey’s three-point error margin. In an ABCNEWS
              telephone poll of 491 registered voters who watched the
              debate, 42 percent said Gore won, 39 percent said Bush
              was the victor, and 13 percent called it a tie. However, the
              difference is within the poll’s margin of error of plus or minus
              4.5 points. 

 
Excerpt from CBSNews web site at
http://cbsnews.cbs.com/now/story/0%2C1597%2C238199-412%2C00.shtml
 

Presidential debate watchers came away with the impression that Vice President Al Gore won the debate against Texas Governor George W. Bush. Over half - 56 percent - of debate watchers feel that Gore won the debate, while 42 percent think Bush did the better job. 

Both candidates improved their image somewhat with voters through the debate. Thirty-five percent of those voters who watched the debate say that their image of Bush has changed for the better as a result. Gore saw a similar gain in post-debate popularity, with 32 percent saying their image of Gore had changed for the better as a result of the debate. Roughly one in five voters who saw the debate say they have lowered their opinions of each candidate as a result.

Bush made only marginal progress, however, in another key area - the perception that he is not prepared to be president.

In the post-debate poll, 54 percent said Bush was well prepared for the job of president. This shows only slight improvement from a CBS News/New York Times poll earlier this week in which 49 percent of voters said they thought Bush was well prepared for the job. In contrast, seven in ten voters said both now and earlier this week that Gore is well prepared for the job.

Vote preferences have changed little as a result of the debate. In a pre-debate survey, Gore had a four-point edge over Bush, 46-42 percent. Among debate watchers, Gore gained a few points, but Bush held his own. Fifty percent of debate watchers support Gore, 42 percent support Bush.

This CBS News poll was conducted online by Knowledge Networks among a nationwide random sample of 812 registered voters. Knowledge Networks polled a sample of registered voters in its household panel, which is a nationally representative sample of households given access to the Internet via Web TV. The margin of sampling error is plus or minus 4 percentage points for results based on the entire sample, and 4 points for results among those who watched the debates.