Based on articles published in the Globe and Mail and on the Market Research and Intelligence Association (MRIA) blog, this extended 4-part series looks at what’s wrong with political polling in Canada (and elsewhere) and asserts that it can and must be fixed. Drawing on his own experience in both the political and market research arenas, and from his interviews with thought leaders and pollsters from across Canada and the US, Brian F. Singh critiques conventional polling methods that are perpetuated by pollsters and passed on to the public by the media, and concludes with a 5-point call to action for the market research industry.
* * *
Part 4 – Where do we go from here?
Just throwing money at this problem is not a solution. The money, frankly, is not there. Costs matter. We are in a new world – low voter turnouts, multiple communication technologies, social media platforms, and the use by parties of geo-demographic targeting and sophisticated voter identification methods to find supporters. These have dramatically affected the political polling business, and pollsters have been slow to adjust and/or they are not evolving their skills.
This is a cultural problem – one of our industry, the media and public engagement connected to and part of our political ecosystem. Grenier is pointed in his summation:
“I think the next time around there will be a lot more reticence to get more than a passing kind of mention of what the poll is showing, rather than using it as a basis for entire articles. I think once an election comes around where the polls do well, and there will be, some of that trust will be regained but it won’t be the same for at least a couple of years I would say.”
Polling is evolving, and has to continue to evolve. The horse-race dimension is damaging our reputation and we are losing the public’s trust. Corporate leaders are asking questions about the accuracy and quality of our work. Thoughtful, more transparent polling is what is being asked of the industry.
As outlined above, we have numerous and emerging realities. And it is only going to get more challenging. Emerging trends will be amplified: new players will always enter the market seeking to build their reputation by giving away their findings, and aggregators (serving as third party evaluators) will likely become our spokespersons. However, I believe that this is good time to reflect and set a firm course of action. This can also be MRIA’s time to shine and provide leadership on a very public issue.
My apologies for the Nate Silver love fest. While he a free-rider in his use of polls, he has made them sexy in the minds of the public. And he has the platform of the NY Times, his book and speaking tour to do it.
Based on my review of practices in other jurisdictions, feedback from interviews with thought leaders, and from my own observations, I propose the following points for our association, and for the polling ecosystem, to consider.
1: Focus on Quality Control I believe that we need to focus on diligence, transparency and disclosure. While many are already diligent, we need to pay greater attention to stratified and structured samples. And we need to be transparent about how data collection and analysis were undertaken. Further, a full disclosure of data collection, including sample sources and field protocols, weighting schemes and, if it was part of an omnibus survey or if the poll was commissioned, needs to be posted. Datasets should also be available for review and, ideally, subject to ongoing academic review. Integral to this is more nuanced polling – moving beyond the horse race and building stronger data integrity.
2: Media Disclosure MRIA needs to immediately establish more stringent reporting standards and work with Canada’s print, electronic and digital media outlets to adopt and enforce them. There are many examples out there – grab them, take the best ideas, and make sure the media adopt them, too. And don’t let up – keep posting polling articles and what was disclosed.
3. Oversight In times of crisis, other jurisdictions undertook inquiries into their polling industry. While MRIA may lack the clout to do this, we can provide leadership. While it would be great to set up a Canadian Association for Public Opinion Research (CAPOR?), it might be more realistic to become a national chapter of AAPOR. Adoption of their resources, protocols and dissemination of education resources, within a Canadian context, would be a positive step.
4. Establish an online information database to inform polling: I believe the onus is on us to collect, review and triangulate as much data as we can before we design a poll. MRIA could consider establishing a “pay to play” central database that pulls in and organizes all political data from social media platforms – e.g., hashtags (such as #cdnpoli, #bcpoli, #ableg, #onvote), blogs, articles (e.g., media clipping service) and other types of commentary – that can be analyzed to assess emerging issues and trends that can be used to inform polling questions. Establishing a forum to discuss how these data are used in questionnaires and methodologies could serve as a valuable complement. A potential partnership with The Hill Times? Could this be an opportunity for app development? Canadian political geeks will be all over it.
5. A Real Experiment: While this may be a call to return to first principles, I think an in-depth project with real-time research on research is required. I propose that our industry look at the 2015 federal election, and work with media and academics to collaborate and co-create an introspective, future-oriented national polling project. It can be multi-modal, but more importantly, we can build on inventory of insight and dialogue (using YouTube, Google/hangouts and podcasts) on preparing for and polling during an election. Aspects such as A/B testing, broad population versus voting populations, and analysis of swing ridings only, could be conducted. Aggregators, who benefit from our work, could be brought in to provide another critical perspective. MRIA should coordinate this.
While the emergent media/pollster business model requires careful examination, the current business model of the media overrides any quick resolution of the “fast and cheap” polling problem. I have stopped short of advocating for only publishing polls that are paid for, or for labeling free poll results as “advertorial.” While this is going to take a lot of money (I am not delusional about this), it is our reputation and trust in our industry that is at stake.
MRIA should approach all levels of government, foundations and think tanks to seek out the funding to pursue these recommendations (including possible use of SR&ED and IRAP grants). The rest of the funds can come from industry, with substantial sweat equity, and finally from the media. Ultimately, the project will be transformative and will serve as a LEAN review of our industry and ecosystem.
Poor polling is a symptom, not a cause, of weak voter turnout. While voter turnout will continue to plague our elections, at least we can begin to put to rest any problems and beliefs associated with suspect polling and its subsequent reporting.
* * *
American Association for Public Opinion Research: http://www.aapor.org/Home.htm
Associated Press Stylebook on polls and surveys: http://ralphehanson.com/blog/ap_poll.html
The New York Times Polling Standards: http://www.nytimes.com/ref/us/politics/10_polling_standards.html
BBC Opinion Polls, Surveys, Questionnaires, Votes and Straw Polls – Guidance in Full: http://www.bbc.co.uk/editorialguidelines/page/guidance-polls-surveys-full
Nate Silver – Which Polls Fared Best (and Worst) in the 2012 Presidential Race: http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/
CBC, The Current – The Power of Polls (May 16): http://www.cbc.ca/thecurrent/episode/2013/05/16/the-power-of-polls/