Sam Wang of Prince Election Consortium: Why State Poll aggregates are likely more more accurate
than National Poll aggregates:
Ro-mentum watch: Karl Rove
by Sam Wang, November 1, 2012, 9:10am comment thread
In this year's race, national polls show a tie, while state polls show a decisive Obama advantage. Here I suggest that the difference may arise from the fact that the same systematic pollster errors can have different effects depending on whether they occur in national vs. state surveys. Based on past elections, national poll aggregates differ from election results by as much as 2.5%. During the same period, state-poll aggregation has been considerably more accurate. Even if state polls have the same accuracy as national polls, races at that level are usually decided by larger margins, leaving room for aggregation to remove the effect of the error. I suggest that the Meta-Analysis of state polls provides a more accurate poll-based prediction of next Tuesday's outcome than national polls.
In the Wall Street Journal, Karl Rove surprises basically nobody by predicting a Romney win. His reason? He cites a Romney lead in some national polls. This has become a rallying cry for the right. But is 'the math' correct?
Here at the Princeton Election Consortium, the Meta-Analysis of state polls points toward an Obama electoral victory. The median outcome is Obama 308, Romney 230 EV, with a Meta-Margin of Obama +2.4%+/-0.5%. To put it into plain English: If state polls are accurate on the whole, then Obama will win.
However, national polls give a different result. National polls since October 14th give a tied median, 'Obamney' +0.0 +/- 0.3% (n=44 polls, median +/- estimated SEM). Indeed, the discrepancy with the Meta-Analysis has been over 2.0% all season.
What is going on? Nate Silver chewed it over yesterday. Let's go through some possible reasons using PEC's approaches.
Do differences in national and state poll methods account for the discrepancy? If we only accept polls from organizations that survey both the national race and individual states, we will have an apples-to-apples comparison. The result is the same: a national poll median of Obamney +0.0 +/- 0.6% (n=10 pollsters, 1 poll per organization). Dropping automated phone polls (PPP, Rasmussen, Gravis) gives Obama +0.5%, still not enough to account for the difference. Answer: no.
Are state polls slow to catch up? State polls take 10-12 days to reach a new steady state, even when the change occurs in one day, like Romney's 5-point bounce after Debate #1. Could it be that they have not caught up with national polls? This is unlikely for two reasons: (a) In national polls, the race has been stable for the last two weeks - long enough for state polls to catch up. (b) The Meta-Analysis is moving toward Obama - opposite to the direction expected. Answer: no.
Are there hidden advantages in non-swing states? Unlike state polls that influence the Meta-Analysis, national polls sample non-swing states. Could Romney have exceptional support in red states -- or make the race close in blue states? Using polling margins from Pollster.com (and filling in a few missing values using 2008 returns), an average (weighted by 2008 turnout) gives Obama +2.1 +/- 0.6%. Sean Trende of RCP has done a similar calculation. That is basically the same as the Meta-margin. Answer: no.
How is the track record of national polls? Here is a comparison of poll margins and final results.
Year Final polling median Actual result Discrepancy
2008 Obama +7.0 +/- 0.9 % (n=15) Obama +7.3% 0.3% (0.3 sigma)
2004 Bush +1.0 +/- 0.5 % (n=13) Bush +2.4% 1.4% (2.8 sigma)
2000 Bush +2.0 +/- 0.9 % (n=15) Gore +0.5% 2.5% (2.7 sigma)
For a bell-shaped curve, the average error is supposed to be 0.8 sigma. Here it's much larger, 1.9 sigma. Aha...here may be our culprit. Evidently, national polls have systematic problems. Answer: national polls do about 2.5x worse at predicting the popular vote outcome than expected if the wisdom of crowds of pollsters were perfect.
How is the track record of state polls? In terms of predicting both state-by-state and overall electoral outcomes, state polls do extremely well. In 2008, I correctly identified the leader 49 out of 51 races. I called two races (Indiana and Missouri) tossups, and those races had margins within 1%. In addition, the 2004 EV median precisely matched the final outcome. In other words, state polls get it 98-100% correct. Answer: pretty darned good.
But if state polls use the same methods, why would they do better than national polls? Well, state polls have three advantages. (1) Most state races, even in swing states, are decided by margins of 2% or greater. So an error that makes a big difference in national polls doesn't matter nearly as much for state polls. (2) State polls target more homogenous populations, which poses fewer technical problems to the pollster. For this reason, the systematic error might be smaller. (3) In critical swing states they are done more frequently. This focuses the data where information is most needed.
As for why the weighted sum of state polls gives a result that differs from national polls, the only reason I can think of is (2) above: state polls might be technically easier to conduct and weight. Still mulling that one.
BOTTOM LINE: Even if national and state polls have the same flaws, they are consistent with one another. Because state poll aggregation is so powerful, the result based on state polls is likely to be more accurate. That is what I would call The Math.
http://election.princeton.edu/romentum-rove-1nov2012.php
