Survey question wording: Lessons learned from same-sex marriage

August 31, 2015
Share Button

The historic Supreme Court decision in late June opened marriage to same-sex couples across the country. Same-sex couples can now marry in all 50 states and the District of Columbia. In addition, those already married can travel the country knowing that their marriage must be recognized equally in all states.

Public opinion and same-sex marriage.

Over time, a major factor in policy debates for or against same-sex marriage was the level of support or opposition present within the general population. Where was public opinion situated with respect to this policy? Of course, this differed by region, by state and by population demographics.

Recent national polls report that the majority of the public supports marriage recognition for same-sex couples. But these reports are not uniform.  Some polling organizations still report less than a majority in favor. So, what is going on? Why the variation?

In order to examine the differences in the survey results, Andrew R. Flores, our Public Opinion & Policy Fellow, put together a dataset of 138 national polls since 1996 (for more on his sampling strategy, go here).

Question wording on public opinion surveys.

It is widely understood among scholars and polling agencies that when designing a survey, question wording matters. Popular pollsters at Pew Research Center say that “the choice of words and phrases in a question is critical in expressing the meaning and intent of the question to the respondent.” And that “even small wording differences can substantially affect the answers people provide.”

While some may view question wording as an art, Dr. Flores takes a scientific perspective on the subject. In his recently published paper, one of the factors he focused on in his analysis was the difference in question wording when asking opinions about same-sex marriage.

Among the 138 polls, he found 36 different ways of asking about same-sex marriage. 22 of them involved a difference in question wording. We present each question wording and the corresponding random effect size on reporting support for same-sex marriage in the figure below.

F5.large

So, was the difference in wording the reason for the differences in reports of public support?

Not really.

He found that there were minimal differences depending on question wording– notice that each random effect has an uncertainty interval (the “whiskers”) that crosses zero.

Ultimately, the variation reported by different question wordings was mainly accounted for by whether:

(1) a neutral response option was provided (such as “Don’t Know”);

(2) whether only two response options were provided (such as “Favor or Oppose”); and

(3) whether the questions focused only on the issue of “same-sex marriage.”

These findings can have important implications for survey design going forward, adding a little more support for the scientific perspective of survey question design. Beyond same-sex marriage, the methods provided in Dr. Flores’ paper may be used to examine other issues where there is variation among reports of popular opinion.

Share Button

Data freely available at http://poq.oxfordjournals.org

The analysis relies on surveys that were obtained from searches of the Roper Center iPOLL Databank, University of Connecticut, and from reports from PollingReport.com. All polls are of the American adult public and were collected via telephone. The average sample size is 1,323 respondents, with a minimum sample of 443 and a maximum of 4,509.

ABC News/Washington Post (9/03, 1/04, 2/04, 3/04, 8/05, 5/06, 4/09, 2/10, 3/11, 7/11, 3/12, 5/12, 7/12, 11/12, 5/13, 6/13, 5/14), Associated Press (5/00, 11/04, 8/11, 6/12), Associated Press/National Constitution Center (9/09, 8/10, 8/11, 8/12), Bloomberg (5/13, 9/13, 3/14), Boston Globe (5/05), CBS News (7/03, 12/03, 2/04, 7/12, 9/12, 11/12, 2/13, 3/13, 5/13, 7/13, 2/14), Cable News Network (8/07, 10/07, 6/08, 12/08, 4/09, 5/09, 8/10, 4/11, 9/11, 5/12, 3/13, 6/13), Gallup (3/96, 2/99, 1/00, 6/03, 10/03, 12/03, 2/04, 3/04, 7/04, 3/05, 4/05, 8/05, 5/06, 5/07, 5/08, 5/09, 5/10, 5/11, 12/11, 5/12, 11/12, 5/13, 7/13, 5/14), General Social Survey (2008, 2010, 2012), Los Angeles Times(11/03), NBC News/Wall Street Journal (7/03, 3/04, 10/09, 2/12, 12/12, 4/13), National Annenberg Election Survey (2/04, 5/04), National Public Radio (12/03), Newsweek (12/08), Public Religion Research Institute (3/12, 6/12, 5/13, 1/13, 11/13), Pew Research Center (10/03, 11/03, 2/04, 3/04, 7/04, 8/04, 12/04, 7/05, 3/06, 6/06, 7/06, 5/08, 6/08, 8/08, 4/09, 8/09, 7/10, 8/10, 2/11, 4/12, 6/12, 10/12, 3/13, 5/13, 2/14), Quinnipiac (7/08, 4/09, 7/11, 11/12, 2/13, 3/13, 4/13, 6/13),Time (6/08, 7/08), Time/Cable News Network (2/04), USA Today (6/13).

By: Angeliki Kastanis