An excerpt from the Social Science and Statistics Blog:
“How’s it going?” If you ever tried to compare the answer to this question between the average American (“great”) and European (“so-so” followed a list of minor complaints), you hit directly on a big problem in measuring self-reported variables.
Essentially the responses to questions on self-reported health, political voice and so on are determined not only by differences in actual experience, but also by differences in expectations and norms. For a European “so-so” is a rather acceptable status of wellbeing whereas for Americans it might generate serious worries. Similarly people’s expectations about health may change with age and responses can thus be incomparable within a population (see this hilarious video on Gary King’s website for an example).
A way to address this problem in surveys is to use “anchoring vignettes” that let people compare themselves on some scale, and then also ask them to assess hypothetical people on the same scale. The idea is that ratings of the hypothetical persons reflect the respondents’ norms and expectations similarly to the rating of their own situation. Since the hypothetical scenarios are fixed across the respondents any difference in response for the vignettes is due to the interpersonal incomparability.
Using vignettes is better than asking people to rank themselves on a scale from “best” to “worst” health because it makes the context explicit and puts it in control of the experimenter. Gary and colleagues have done work on this issue which shows that using vignettes can lead to very different results than self-reports (check out their site).
…
One fix is to use anchoring vignettes that let the interviewer control the context against which ratings are made.
For example, in a 2002 paper on the use of vignettes in health research, Salomon, Tandon and Murray ask respondents to rank their own difficulty in mobility on a scale from ‘no difficulty’ to ‘extreme difficulty’. Then they let respondents apply the same scale to some hypothetical persons using descriptions like these:
“Paul is an active athlete who runs long distances of 20km twice a week and plays soccer with no problems.”
“Mary has no problems walking, running or using her hands, arms, and legs. She jogs 4km twice a week.”
Using the difference in how people assess these controlled scenarios, one can adjust the rating of people’s own health. Doing this across or within various populations then allows to examine systematic differences across groups. These vignettes have been used in recent World Health Surveys in a number of countries.
King, Murray, Salomon and Tandon introduced the vignettes approach and used the measured differences to correct responses to self-rated questions on political efficacy. The idea is that applying the vignettes to a sub-sample is cheap and sufficient to understand systematic differences in self-reports. Their methods are laid out in the paper, but the results show how much difference the vignettes method can make: instead of suggesting that there is a higher level of political efficacy in China than in Mexico (as self-reports would indicate), the vignette method shows the exact opposite because the Chinese have lower standards for efficacy and thus understand the scale differently.
Intuitively that’s what we do all the time: once you talked to enough Europeans and Americans about their (and other peoples’) well-being you use your mental model to adjust responses and stop taking the European’s minor complaints too seriously. Using this insight in survey-based research can make a huge difference too.