Anatomy of a High Quality Poll (Quinnipiac)

[cthulu2016]

Last edited Wed Oct 31, 2012, 08:51 PM - Edit history (2)

The New York Times published this very open and cogent explanation of the methodology of a poll they commissioned. All pollsters should be equally comprehensive in describing the method of given poll. (Being the fine print on a poll, rather than a news story, I think it is fair use to discuss the entirety of it.)

http://www.nytimes.com/2012/10/31/us/politics/how-the-polls-were-conducted.html

These were the recent NYT/CBS polls that came in Obama +5 in Ohio, +2 in Virginia and +1 in Florida.

How the Polls Were Conducted

The latest Quinnipiac University/New York Times/CBS News polls included a total of 3,682 telephone interviews conducted Oct. 23 through Oct. 28 with adult residents of Florida, Ohio and Virginia. Of these, 3,394 said they were registered to vote.

Results are based on 1,073 likely voters in Florida, 1,110 likely voters in Ohio and 1,074 likely voters in Virginia. Likely voters are adults who say they are registered to vote and will “definitely” vote on Election Day.

The likely voter screen is defined up front, and is simple. A more intricate screen might be more or less accurate, but the significant thing here is that they tell you exactly what the screen they used is. I like that.

The time span of the sample is long (6 days) making it a less crisp "snapshot," but smoothing out potential distortions like weekend vs. week day, some big sporting event, some weather anomaly, etc..

The sample sizes are quite large. The sample size for each state is about the size of most national polls, making the margin of error a bit smaller than most state polling. (I think most state polling base samples are around 800.)

The poll is live interviews, not robocalls.

All interviewing was conducted from the Quinnipiac University Polling Institute in Hamden, Conn. Interviews were in English or Spanish.

The telephone numbers called, both land lines and cellphones, were from random digit dialing samples provided by Survey Sampling International of Shelton, Conn. The regions within each state were represented in proportion to their share of all telephone numbers. Random digits ensured access to listed and unlisted numbers alike.

English and Spanish is good. Most polls are English only.

They tried to get a representative sample of the various regions in the states up front, which is good. (Those are probably refined by weighting after the fact, but optimally your initial sample should require as little weighting as is practical.)

The completed samples were adjusted to ensure the proper ratio of land-line-only, cellphone-only and dual-phone users. Within each household one adult was designated by a random procedure to be the respondent for the survey.

Each state’s results have been weighted to adjust for variation in the sample relating to region, sex, race, Hispanic origin, age, and education.

Both landlines and cell phones were called, and then those samples were then weighted to further correct distortions. And all the usual demographic factors were weighted.

It is nice that they list precisely which demographic categories are weighted.

Interviewers made multiple attempts to reach every phone number in the survey, calling back unanswered numbers on different days at different times.

This multiple call-backs method is a reason the polls were in the field for 6 days. Call backs help avoid polling only people who are easy to reach.

In theory, in 19 cases out of 20, overall results based on such samples will differ by no more than three percentage points in either direction from what would have been obtained by seeking to interview all voters in each of the states. For smaller subgroups, the margin of sampling error is larger. Shifts in results between polls over time also have a larger sampling error.

In addition to sampling error, the practical difficulties of conducting any survey of public opinion may introduce other sources of error into the poll. Variation in the wording, order or translation of questions, for example, may lead to somewhat different results. In projecting final pre-election polls like these, events that transpire after the interviewing and before Election Day may also affect the outcome.

This is a nice explanation of the 95% confidence margin of error range. In this poll it is +/- 3%, which is quite low.

The math and science of random samples is powerful, but counter-intuitive. Asking 1,000 people in Florida really is almost as good as asking everyone in Florida. That's a mathematical fact, and can be quantified precisely. As they say here, there is a 95% chance (19 out of 20) that these results are within 3% of what polling everyone would tell us.

Note the fine distinction made here... not how people will vote, but what people would say if we polled everyone. A poll can only directly measure how people answer a poll. We infer that people who say they are for Obama are actually for Obama in the real world, but the correlation of what people say and what people do is outside the margin of error question.

[ComplimentarySwine]

1. Thanks for your insight

Polling methodology is very confusing to me, although I think that I'm starting to get a handle on it thanks to information such as you provided here. One thing that I wish that I knew more about was how they go about weighting polls for all of the various demographic information.

[cthulu2016]

3. Some are easy, using census data and past exit polling

For instance, the electorate is almost always 51-52% female. The population of a certain block of states is a known quantity. The census is used for ethnic data. (The census is also self-identified, so it's apples to apples.)

The tricky one these days is cell phones. All pollsters have some weighting formula for cell phones versus landlines, but that must be based on a whole range of information because, unlike race or sex, it changes constantly.

[hay rick]

2. Proliferation of polls.

I'm not much of a poll watcher, but it did seem to me that there has been a proliferation of polling companies. I noticed that the "fine print" on several of the polls I saw for local (Florida) races noted that registered Republicans were oversampled. My belief is that many polls are commissioned by partisans to support candidacies and by media organizations to reinforce their horse-race narrative. Donning my tinfoil hat- I think "fact-checking" organizations are subject to the same forces...

[cthulu2016]

4. There are good pols and lousy polls. The reason I posted the OP is

to provide an in depth example of a high-quality poll. High quality polls are expensive, but more accurate than inexpensive robocall polls. (Robocalling tends to get more older people and republicans)

Most of the fly-by-night propaganda polls, such as you describe, are very cheap and you can tell from the methodology how flimsy they are.

[uponit7771]

5. THIS NEEDS TO BE A STICKY

[cthulu2016]

6. .