Stats Rats Needed: Can someone explain significance of variance in the polls?

[kennetha]

So I do my usual thing over my morning cup of coffee and check out the latest polls on RCP.com. And I notice -- just to take one example -- that the polls they currently use to calculate the polling "average" for Ohio are all over the map. Two of the polls have McCain up by tiny margins. Three of the polls have Obama up by huge margins and one of the polls has Obama up by a modest margin.

And I wonder to myself, what really is going on with these polls. Why do they vary so significantly? Is it the model of likely voters that explains most of the variance? The sampling technique? Or what? And assuming that there's a fixed underlying reality that each of the polls is supposed to be measuring, exactly why do these methodological differences make such a big difference? And I also wondered whether averaging really improves the informativeness of the polls. Is the average of a bunch of unreliable measures more reliable than the individual measures?

Here's an analogy. Suppose you had six different thermometers and you took your temperature with each one and each one gave you a different reading. You'd assume you have a fixed temperature and that at least five of the thermometers must be wrong. Would taking the average reading of the six thermometers increase your chances of knowing what your real temperature is? Intuitively, I don't think so. But I wouldn't be surprised if there wasn't some theorem in statistics that justified the averaging.

But wouldn't it be better if there were a way of discounting some polls more than others? Averaging them all treats them all equally -- both the most error prone and the least error prone get counted equally in calculating the average. That doesn't seem like a way to enhance the informativeness of an imperfect measure.

Anyway, I find these wildly divergent polls fascinating and frustrating. If some smart person could explain how best to look at them -- or point to some source that already does that -- I for one would be grateful for the enlightenment.

[speedoo]

1. Couple of points...

Pollsters use different assumptions about makeup of Republicans and Democrats when they do polls. One might use 40 - 40 in Ohio, another might use 42-38. So that will contribute to differences.

(on edit... the "Likely Voter" polls make assumptions about the probability about the likelihood that different demographic groups will actually vote. Using these assumptions, the pollsters then jigger their results up and down. I usually ignore these.)

Another source of differences is the fact that there is always what is called statistical error, due to the fact that pollsters use samples that are very small relative to the total population. The "margins of error" that they publish with their polls addresses that.

A site that usually provies very informative stuff about the polling process is fivethirtyeight.com. Highly recommended.

[kennetha]

2. Right , I get that much

But the thing I don't get is given that the polls are so sensitive to the setting those parameters -- as evidenced by the huge variance -- how exactly are we to take any of the polls or even the aggregate of them. I'm especially curious about the significance of the average of a set of polls that are kind of all over the map. In particular, does the average of the polls in any way improve informativeness? I don't really see why it should.

Wouldn't it be more informative to look at how outcomes change systematically as you vary various things -- like the party id make up, the likely voter screen, etc By looking at that, it seems that maybe you could figure out at least what the outcome of the election hinges on. That sort of information might be more useful than just information about the distribution of "point" preferences of voters at a given time under a given set of assumptions.

If I was a candidate, I guess I'd want to know how my chances changed under different assumptions about turnout, party id distribution, etc. Cause these are things that I might be able to drive in my favor.

[Hamlette]

3. Nate has been studying and explaining each poll and each state all year

you could go back and read his archives. He's a statistician so I get the impression he knows what he's talking about.

www.fivethirtyeight.com

there are lots of factors and explains the importance of each one, demographics is pretty high on the list

[kennetha]

4. Thanks

For some reason, I wasn't aware of 538 until just the other day when a friend mentioned it.

[Hamlette]

5. I live there

going there makes me not so scared...'cept I didn't go there when McCain has his post convention bounce.

[DevonRex]

6. That AP poll you posted the other day DOUBLED the number of

evangelicals from 2004. In 2004 it was something like 22%. They predicted 45% (or something like that) for the 2008 race.

Nobody on earth believes that evangelicals make up almost half of the population.

That's just one example of the way the predictors, which are chosen by the poll company, can skew a poll.