Why is national polling all over the map?
By Philip Bump October 27 at 2:58 PM
One of the most insightful articles about polling this cycle comes from the New York Times's Upshot team. With raw poll data from Florida in hand, Upshot asked four pollsters to interpret the results. The data each pollster used was the same, and the Upshot's analysis figured that it showed a 1-point lead for Hillary Clinton in the state. Those four pollsters didn't all come back with the same result. The responses were a 1-point Clinton lead, a 3-point Clinton lead, a 4-point Clinton lead and a 1-point lead for Donald Trump.
The point of the exercise was to highlight that pollsters make certain assumptions about the data they get back from their interviews. They make assumptions about who is likely to vote and about how to weight the data, for example, which is a different consideration than the quality of the responses you've gotten from talking to voters.
RealClearPolitics's Sean Trende pointed to the Upshot's results in a series of tweets Thursday morning looking at recent national poll results. “The spread on the polls right now really is bothersome,” he wrote, suggesting a lot of different assumptions about what the electorate will look like. With Clinton's lead as big as it is, there's not much question about the final result. If the race narrows, those varying assumptions could be a problem.
Since Oct. 20, a week ago, there have been at least seven major national polls, including the Investor's Business Daily tracking poll (for which we'll look at the most recent iteration) and our own tracking poll conducted with ABC News. Clinton leads in each; you can see them at the RCP website. Each poll was conducted over a different time period, so we've plotted the results — red for Trump, blue for Clinton — across the days the polls were conducted.
You can see the messiness. Clinton's anywhere from 43 to 51 points on Oct. 22 and 23; Trump ranges from 34 to 44. And that's before we add the margins of error.
Read more and see more graphs: https://www.washingtonpost.com/news/the-fix/wp/2016/10/27/why-is-national-polling-all-over-the-map/
= new reply since forum marked as read
then the odds of any single poll falling within 1/4 of the MOE of the "true" support for a candidate is about 20%, or 1-in-5. That's not very high. If you were able to take dozens or hundreds of polls using the same methodology then they would form a distribution around the "true" result such that 68% of the polls would be within the MOE (on each side). 95% of the poll results would be within 2 times the MOE on each side of the result. If the MOE is 3% then that translates to a range of 12% (±6%).
The average of the polls in the graph above for HRC is 46.4%. If the MOE is 3.5% then 65% of the polls should be between 42.9% and 49.9%. 5 out of the 7 (71%) are in this range and the other 2 polls are within 2 times the MOE. This suggests that the spread of poll results that we see is entirely consistent with random variation due to sampling. If we could do a lot more polls, you would expect them to form a symmetric distribution with a peak near the current mean.
I think the bottom line is don't read too much into a single poll result when there are other polls available. If the polls are all using good sampling methods then you'll get a much more accurate result from averaging them together. That's why when the media touts a particular poll to the exclusion of others it's more than a little dishonest, in my view.
If you want to find out more about normal distributions, Wikipedia is helpful: https://en.wikipedia.org/wiki/Normal_distribution
Here's a graph that is illustrative:
