The rap on research for the arts, museums, and informal sciences
In a recent lunch conversation here at Slover Linett, my colleagues and I discussed how social desirability bias affects our research work and what we can do to minimize its impact.
As research & evaluation consultants working in the arts and culture field, we try to understand people’s attitudes, values, behaviors, and beliefs. We often ask people to tell us about themselves and their experiences through a variety of methods: surveys, group discussions, interviews, ethnography, and all kinds of creative sessions and facilitated exercises. But many people find it difficult to access and accurately share their thoughts and feelings at the drop of a question. They tend to answer research questions the ‘right’ way — that is, instead of telling us how they really feel or behave, they answer in a way that they think is socially acceptable. This tendency can leave us with a false, idealized version of how people think and act. That’s social desirability bias, and it’s a universal challenge in social research.
This is a very complex issue. Some participants answer researchers’ questions in a socially desirable way on purpose, for fear of being judged. Others may not be aware that they’re doing it at all. To better understand what influences social desirability bias, we need to further unpack these ideas.
Part of what’s going on here is about the core of what makes us human: the need to be socially connected to others and the desire to belong to a group. Being socially connected to a group was evolutionarily important for our sheer survival, and these days it is still essential to our health and happiness. Part of what holds a group together is a shared set of beliefs, values, and behaviors — let’s call them social factors. For every group, these social factors are placed on a scale of desirability, from the taboo to the highly prestigious. The social factors on the highly prestigious end of the spectrum are where people may want to be, the ideal version of themselves that they want to see in the mirror. When people answer research questions from the perspective of this ideal self, social desirability bias is in play.
This is even more common in the cultural sector, where my colleagues and I do our research and evaluation work. People often ascribe an elevated level of status and prestige to arts institutions and museums of all kinds. That’s part of what it has meant, traditionally, to belong to the group ‘educated, upper-class, civilized Americans.’ Those beliefs are strong enough that people have a tendency to say that arts and cultural institutions are extremely important and praiseworthy even when their own needs as audience members are not being met. Instead of blaming the institutions, they often blame themselves. This bias can tilt their answers to researcher’s questions in favor of the cultural institution, inflating satisfaction ratings and other indicators of value and success.
Social desirability bias can also be understood through self-perception theory. Developed by Daryl Bem, the theory states that we continually interpret our own behaviors in order to construct a sense of self. Rather than having a pre-existing, coherent identity that causes us to act in certain ways, we observe our own actions and decide on that basis who we are. But if our attitudes and motivations are really just after-the-fact rationalizations of our behaviors, how can social researchers ever truly discover what makes people tick? It’s a truly puzzling problem, and over lunch we got pretty philosophical about it.
But the practical challenges social desirability bias poses for us are very real. If audiences are prone to give idealized responses to our questions about cultural participation, and if they don’t readily have cognitive, conscious access to the values and desires that motivate them, then we need to use research methods that dig deeper. Here are a few ways we try to minimize the effects of social desirability bias in our studies for museums and arts organizations.
Designing good questions. As researchers, we put a lot of effort into designing questions that minimize the possibility of social desirability bias. If we have the time, and more importantly the budget, we often suggest conducting a round of cognitive testing as we develop the questionnaire. In cognitive testing, we ask people to think out loud as they read and answer each survey question, to see how they understand what we’re asking and how wording influences the way they respond. Those discussions give us a chance to see where and how social desirability bias may be cropping up.
‘Challenging’ their answers. In order to help free respondents from the ‘right answer’ trap, we often, challenge or contextualize their self-reported attitudes byasking about the opposite attitudes and behaviors to see if they resonate. Let’s say you’re interested in understanding motivations. In addition to listing the ‘desirable,’ familiar motivations as possible answers — such as ‘learning’ or ‘exposing my children to something new’ — consider including less prestigious-sounding options, such as ‘getting away from my everyday routine,’ ‘impressing my date,’ or ‘because someone else dragged me along.’ This is also about question order: If there is an easy or obvious response (like, ‘free admission’) place it as the very last option. This will force the respondent to consider all the choices before they reach for the easy or obvious response.
Let past behavior be a proxy for present attitudes. One of the most effective tools for getting around social desirability bias is to ask respondents about their past actions. After all, what we’re interested in is usually some kind of behavior — participating, joining, subscribing, supporting, recommending, or taking some civic or conservation action. All the attitudes, values and beliefs in the world don’t matter much unless they’re accompanied by action. And nothing predicts those behaviors better than whether someone has done them in the past. By probing into how people have actually behaved in the past and what influenced those behaviors, we‘re able to get a better understanding of who an individual truly is and how they are likely to act in the future.
Triangulate with multiple methods. Perhaps the simplest technique we discussed to mitigate social desirability bias is using a mixed-method approach: incorporating both qualitative and quantitative techniques in a single research study. By comparing the qualitative responses with the survey data, we can triangulate richer insights and get a sense of how candid respondents are being with us — and with themselves.
Social desirability bias is a complex problem in social research, especially research in the arts and informal learning. There are many different ways to think about it and mitigate its effects; the ideas I’ve presented here are by no means exhaustive. If you’re a fellow researcher or evaluator, I’m interested in hearing how you think about it and what tools and tips you use to manage it. Please share your thoughts below in the comments.
0 Comments »