Nate Silver, Gravis Marketing, and some Smoking Guns
Many thanks to Grantcart, we now know that one of the pillars of Silver's polling scheme is a right-wing fraud with no background in polling. But fivethirtyeight continues to include the Gravis polls in its data. They say they only exclude polls that meet a "quite narrow" definition of being partisan.
But shouldn't they also exclude polls that don't meet minimal industry standards for polling organizations? Polls that, for example, use undefined “likely voter” screens and undefined weighting according to undefined categories of voters?
Gravis Marketing includes no details about the results of its political polls on its website; and no information about methodology on either its website or its polling result blog.
Website: http://gravismarketing.com/
Blog: http://gravismarketing.blogspot.com/
All of the services listed on its website are for political campaigns, including help with GOTV efforts. (Apparently, Nate Silver doesn’t see any possible conflict of interest with this.) So all we have to go on are the press releases and the blog. Here is an example: the press release for a poll supposedly conducted in Missouri in September. (Which I assume can also be found on the blog, though I didn’t look for it there.)
http://www.realclearpolitics.com/docs/2012/Gravis_MO_0917.pdf
According to the poll, Gravis conducted a “telephone survey of 1,959 likely voters in the state of Missouri on the afternoon and evening of September 16th and 17th, 2012.”
Who are “likely voters,” according to Gravis? They could be anyone – Gravis doesn’t explain this either in the news release or on his website. Was the survey a truly random survey? It doesn’t say. Did it involve robo-calls or were there human interviewers? It doesn’t say. Did it include cell phones? It doesn’t say. What was the response rate? (A key factor in determining reliability of a “random” sample.) It doesn’t say.
(If you are willing to pay to read the complete report of the survey, maybe you’ll find this information there – or maybe not.)
What DOES the press release say? Among other things:
“The results were weighted by the anticipated voting groups in the upcoming 2012 election.”
WHAT THE HELL DOES THAT MEAN? We don’t know, because they don’t tell us.
The bottom line: We don't know how the survey was conducted (whether the surveyed group included cell phones, what was the response rate, etc.); and we don't know what the “likely voter” screen consisted of, and how the voter groups were weighted. Without this methodological information, this data is worthless.
Here is the information the major polling industry organizations says should be included in the reports of all political surveys, according to a University of Michigan report:
http://www.isr.umich.edu/src/smp/Electronic%20Copies/91.pdf
Obligations to the Public. The AAPOR Code of Ethics attempts to satisfy the profession’s obligations to the public by mandating the disclosure of minimal information about
a poll or survey whose findings are publicly released. The elements that must, at a minimum, be disclosed are the following:
1. Who sponsored the survey, and who conducted it.
2. The exact wording of questions asked, including the text of any preceding instruction or explanation to the interviewer or respondent that might reasonably be expected to affect the response.
3. A definition of the population under study and a description of the sampling frame.
4. A description of the sample selection procedure, making clear whether respondents
were selected by the researcher or were self-selected.
5. Size of sample and, if applicable, completion rates and information on eligibility
criteria and screening procedures.
6. A discussion of the precision of the findings, including, if appropriate, estimates of
sampling error and a description of any weighting or estimating procedure used.
7. Which results, if any, are based on parts of the sample rather than the entire sample.
8. Method, location, and dates of data collection.
Together, these items of information communicate some idea of the error likely to be associated with estimates from a given survey, or, conversely, the degree of confidence warranted by the results. The Code requires that this information must be included in any report of research results, or, alternatively, that it be made available at the time the report is released. In other words, researchers are not required by the Code of Ethics to follow the best professional practices, but they are required to disclose the practices they used.
= new reply since forum marked as read
