A paper by Alan Krueger and David Schkade discusses the reliability of self-report measures of well-being.

ABSTRACT

This paper studies the test-retest reliability of a standard self-reported life satisfaction measure and of affect measures collected from a diary method. The sample consists of 229 women who were interviewed on Thursdays, two weeks apart, in Spring 2005. The correlation of net affect (i.e., duration-weighted positive feelings less negative feelings) measured two weeks apart is 0.64, which is slightly higher than the correlation of life satisfaction (r=0.59). Correlations between income, net affect and life satisfaction are presented, and adjusted for attenuation bias due to measurement error. Life satisfaction is found to correlate much more strongly with income than does net affect. Components of affect that are more person-specific are found to have a higher test-retest reliability than components of affect that are more specific to the particular situation. While reliability figures for subjective well-being measures are lower than those typically found for education, income and many other microeconomic variables, they are probably sufficiently high to support much of the research that is currently being undertaken on subjective well-being, particularly in studies where group means are compared (e.g., across activities or demographic groups).

The potential issue with a correlation of .64 is that is made up of many components besides reliability. That is, the test-retest correlation is thought to get at “repeatability” or “consistency.” However, in “state-like” variables, we would expect there to be a large change component over a two-week period. An analogue would be measuring weight, we want the scale to be reliable or consistent, but over a certain amount of time, my weight is expected to change. The more possibility for change there is in the variable, the more likely the reliability coefficient will be lower.