By Cory Casey and Chris Hubble
Asmarket research professionals, we’ve spent a collective 40 years in the business of gathering, listening, and interpreting the amazingly diverse opinions of Americans and their fellow humans around the world. Above all else it has been a passion, a joy, a constant surprise, and an experience rich with learning and illumination. But it’s also a serious business — whether working for multi-national consumer research companies, building and selling hundred million dollar companies, or advising some of the worlds biggest brands, such as Microsoft, Hewlett-Packard, McDonalds, eBay, Facebook, Google, and Conoco-Phillips — there is a lot at stake. So when we see research being so blatantly misused we feel compelled to call it out for what is known in our line of work as the ultimate Cover-My-Ass-move.
While the political and academic debate rages on about whether or not America is more divided than ever, it should be abundantly clear to everyone that we should not be looking to politicians to bring us back together. Today’s crop of politicians are products of the system, and while there is increasing agreement across the aisle that the system is broken, there is little vision for how to fix it. Instead politicians appear focused on lighting a fire under our destructive differences versus our utilizing our constructive differences. Differing perspectives coming together tend to contribute to the best and most balanced solutions and outcomes after all.
Take for example a recent poll issued by the offices of President Trump and Vice President Pence. The poll so clearly set out not to learn but instead to confirm what the administration already believes. To prove they were right and anyone who disagrees with them wrong. While there have been promises to make America great again, this poll clearly indicates that for this administrationbeing right is everything and controlling the political dialogue is a critical goal. It seems getting data to prove they are right is the priority. This is doubly demonstrated by the very existence of this survey, which is a reissue of the first survey, which was replaced because it was “…Sabotaged by Liberals.” Furthermore, we continue to see the line between campaigning and governing blurring even more. A POTUS who ran on being a candidate of action is instead making never ending tit-for-tat attacks a permanent part of civic life, which in turn results in government paralysis.
The future we want, “…Will only happen if we fix our politics,” said President Obama in his 2016 State of the Union address. He continues, “If we want a better politics — and I’m addressing the American people now — if we want a better politics, it’s not enough just to change a congressman or change a senator or even change a president. We have to change the system to reflect our better selves.” President Obama was also a product of the system, and he did not have an answer for fixing it. And if politicians are relying on input from surveys like the one here, or the survey prior, then we should not hold out hope that our leaders are acting to heal the American divide.
Survey creation is a serious art form. Doing it inappropriately can affect big decisions — depending on the stakeholder — and in some cases, it could alter history. That’s why the Trump administration’s attempt at polling their voters caught our attention. It was… more than a bit leading. Let’s put it this way, if an employee wrote this survey, it would be a fire-able offense.
The survey leaves no doubt it was intended to create a narrative: the media is “BAD!” and that as a collective, they are wrongfully fighting President Trump. The survey flow unfolds like a Netflix original series, building you up through the penultimate episode, leaving you — the survey taker — emotionally raging by the season finale (a donation page to fight the media.) They wrote this survey to fire up the base and to receive donations. Fine.
Let’s just consider where the survey goes painfully wrong and how we’d improve it. To do so, it’s important to know the core principles that guide us in designing surveys:
- Don’t let survey takers know who’s asking the questions.
- Don’t let survey takers know why you’re asking the questions.
- Don’t let the questions produce response bias.
- Be crystal clear.
So with that, let’s dive right into it…
Rule 2 is broken. The “media is bad” narrative is set — we know exactly why we’re here. The question itself is terrible, unless one wants to lead off a survey with such a pointed, leading question.
Rule 3 broken. Instead of prompting the respondent with “Unfairly” in the question text — go broad in the question and give respondents more answer choices. For example, very fairly, somewhat fairly, bigly unfairly, etc.
Rule 4 broken. Clarity. How does a respondent interpret “mainstream media?” Unless you’re familiar with Trump’s talking points, you may not understand what exactly that’s referring to, and the President’s definition of what this is tends to change, adding to the lack of clarity. Typically we would add a clarifying point, “And by mainstream media, we mean…”
Core principles aside, what a way to kick things off with the first question. Where is the screener? In most surveys, we want to know who’s taking it. Who are they? Where are they from? You might even consider asking political affiliation. If they only wanted to speak to Trump supporters (which it sounds like they did,) the best approach would have been to respect Rule 1 and make it an anonymous survey, not branded Trump — Pence, screening for people who supported him in the election.
Rule 3 broken. This is a perfect example of a leading question — you’re implying the media’s doing a poor job covering the administration (with at least one of these issues.) They didn’t even give an option to select “none of the above,” forcing you to select one. What if you supported Betsy Devos and wanted to only select education… you’re out of luck.
Rule 3 shattered. Here is where the survey takes a biased turn for the worse. Imagine someone had heard about the executive order, formed their perspective on it, and now was presented with a completely different way of perceiving it. They’re even forcing survey takers to link the order to “Radical Islamic terrorism.”
As we stated previously, traditional surveys are intended to understand and capture people’s current perceptions, not artificially build new ones. Isn’t that what campaign ads are for?
To improve the question we would recommend breaking it out into two separate questions:
a) How would you describe the President’s Executive Order?
b) How do you feel the media is reporting on the Executive Order?
Rule 3 Broken. It is too leading. Instead of sharing the results of the poll in the question, it would have been better to ask survey takers what they had previously heard. Whether a majority was for the order, a majority was against the order or it was an even split.
Rule 4 Broken. The question is unclear. Either you are aware of the poll or you are not. To give “No opinion” as an option doesn’t make sense because the question they ask is really whether awareness is there or not, it’s not about opinion.
You could then follow that question up prompting “Whether you knew about it or not, a majority of Americans support the ban.” You would then ask survey takers whether that information changed how they felt about the executive order. Options could be: It makes me support it more, less, the same.
Rule 4 broken. Clarity. There’s too much going on here as you’re asking a double barreled question again. There are multiple issues in one question.
First, you’d want to ask whether respondents felt political correctness had any impact on news coverage. Then, among those that did feel there was a link, you would ask how that had an impact on illegal immigration and radical islamic terrorism, separately.
Rule 4 broken. Clarity. Read it again. It’s messy. Not to mention, it is extremely leading, breaking rules 2 and 3. It makes the bold assumption that Democrats are obstructing Republicans, implying Republicans have never obstructed Obama, and planting that as a fact. This is simply not a question, but a statement of a narrative that they want validated.
Suggestion to improve: don’t even ask the question.
Rules 2 and 3 broken. The response bias here is particularly painful. You’re implying the media is creating noise and that it’s in the President’s best interest to deliver the message directly to the people. Remember, we’re trying to learn what survey takers currently think and feel — not tell them how to do so.
A simple fix here would be to drop “The media’s noise” and ask how they felt about the President attempting to deliver his message straight to the people.
Rule 2 broken. This final question neatly ties up the survey’s running narrative: mainstream media is bad and President Trump or the Republicans should rightfully attack it — what better way to transition to the donation page.
If you want to protect the integrity of the research, it’s likely best not to ask the question, or at the very least drop the word “resources” if the very next screen is a donation page.
At the end of the day, it is clear President Trump’s survey was utilized to motivate respondents to take action and to further validate his belief that mainstream media — whatever that is — is bad. They likely accomplished their goals, but left us survey professionals cringing. So much so there are already quite a few articles about it.
If you’re looking to really understand how people feel, make sure to keep the four key survey principles in mind, as well as these additional recommendations:
- Don’t let respondents change their previous answers as they go through the survey.
- Only show one question per screen so the respondent is focused in on that question and nothing else.
- Avoid using yes/ no/ no opinion answer options — put serious thought into your answer options, ensuring you’ve covered all angles a survey taker could be thinking about.
In summation, simply confirming what one already knows is not informative, insightful nor leadership. It is an addiction to being right. There appears to be little end in sight to either party using the government as a bludgeon. Gone are the days when republicans and democrats reasonably debated diverse points of views in order to settle on what is best for the country, which typically requires compromise on both sides. Today it seems they aren’t even able to put the effort into creating a poll to understand what is important to this country’s citizens, instead focusing their efforts on yet another way to stir up fear and anger in people on both sides.
A well-designed survey is a way of getting accurate information that then leads to well-informed decisions. This administration has demonstrated through these surveys that they are interested in no such thing and are simply too busy compiling inaccurate data through a sloppy survey to cover their asses.
** All images are from the Trump-Pence Mainstream Media Accountability Survey available here: https://gop.com/mainstream-media-accountability-survey/