Why Polls are Increasingly Inaccurate---

[Jackpine Radical]

The short answer is simple: It's getting very hard to draw a random sample of people who will vote.

In this context, the concept of "margin of error" is quite misleading, and is not actually a measure of the poll's range of accuracy. M of E only holds meaning if the poll sample is truly randomly drawn from the population to which one wishes to generalize.

The population in Dem-v.-Bush polls everyone is getting excited about lately is hypothetical--it's the totality of people who would vote for President today if given the chance. The M of E as calculated in instances like this is purely a function of sample size. Increase the sample and reduce the M of E. It's actually a square root function, not a linear one, so you hit a point of diminishing returns over a thousand or 2 respondents. But the point remains, no poll is any better than the sampling technique used to select the participants, regardless of the M of E. Increasingly, as people tend to refuse to answer polls or lie to pollsters, or switch entirely to cell phones, or delist their numbers, the obtained samples do not match the population, and the polls are accordingly corrupted. The single exception is exit polls, which remain relatively accurate because the pollster has access to the entire population of interest, i.e. people who just actually voted. However, exit polls don't see very far into the future at all; they at best allow one to project comlete results on the basis of partial returns.

[Dookus]

1. On what do you base your premise?

"Polls are becoming increasingly inaccurate"

What evidence do you have that that is true?

[randr]

3. The data the polsters work with

does not include cell phone users for one.

[Jackpine Radical]

4. The immediate example I had in mind was

the polling that indicated _______* was going to walk away with Wisconsin. There are other examples, such as the Max Cleland race & other 2002 races. I was just trying to get people from being too carried away with polls that show _______* and _______* beating Bush.

*Candidate names self censored to avoid getting my thread locked.

Edited for typos

[mobuto]

6. Its true

Talk to polsters - they're all saying it. That's polling's dirty little secret - every year it becomes less and less accurate.

99 times out of 100, people called on the phone hang up on you. The ones who answer are not representative of the general public.

[Bandit]

7. A poll said it was true

:shrug:

[BillZBubb]

2. You are correct

But, even given the limitations, the polls taken as a group still have value. They definitely identify trends. They just won't give you a reliable prediction of differences.

[Jackpine Radical]

5. In effect, they are degraded from ratio to interval-level

measurement instruments. I think the problems is even worse than that, though. Their basic validity is debased by lack of correspondence between the sample and the population. Since you can't actually measure the population, you can't be sure how far your sample deviates from it.

[alarcojon]

8. Agreed

So what should we do about it? I think that polls are a negative factor in our electoral process - once a poll is released, this very fact changes the results of future polls. However, given the extent to which polls are used nowadays, they aren't going away anytime soon.

I didn't know that cell phone numbers aren't used in polling - I guess I just never thought about it. Omitting them would certainly skew the sample.

It is still heartening to read these poll results which say that Kerry and Edwards would both beat Bush, precisely for the reason that they convince those on the fence that Bush can be beaten. The vast majority of the population doesn't even understand the statistical margin of error (certainly most journalists don't, judging by the value they impute to changes in polls of 1 or 2 percentage points), much less the very real sampling issues you raise.

Again, what should we do about it?

[central scrutinizer]

9. "Dewey Beats Truman"

Little has changed

[Dookus]

10. yes...

little has changed. The Dewey Beats Truman headline was so rare, it's still talked about 56 years later.

Polls are very accurate. The problem is people use them for the wrong reason. If Kerry is beating Bush 55/43 in today's polls, it doesn't mean he's going to win in November.

polls are very useful if you know what they actually mean.

[Jackpine Radical]

11. The problem, Dook, is

that elections are the only occasions when validating data are collected.

You can say whatever you want about how people would vote today, and (presuming today isn't election day) there's no way to prove you wrong. There is no way to validate a poll like these, and they have therefore never been validated scientifically.