Task sensitivity and noise: How mechanical properties of preference elicitation tasks account for differences in preferences across tasks

Researchers can measure preference using different elicitation tasks. In this article, I propose that the decision how to measure preference is particularly consequential because preference elicitation tasks differ in terms of two properties: (a) their sensitivity to the relative utility of the options and (b) the amount of noise in participants’ responses. I focus on two elicitation tasks: choice and ratings. Through simulations and a series of experiments, I provide robust evidence that a greater percentage of participants prefer “advantaged” options (i.e., options carrying a higher amount of utility) in choice than in rating. This typically occurs because choice is more sensitive to utility differences than ratings and because ratings are sometimes associated with higher levels of noise than choice. Further, I discuss four moderators of the effect: the granularity of the rating scale, the type of judgment elicited by the rating task, the magnitude of the utility difference between the options, and the mode by which alternatives are presented (joint vs. separate).