Inconsistency Problem of Experiment: A Perspective of Attention and Salience

• In Event: Conjoint Design in Focus: Practical and Theoretical Advances

Abstract

The randomized experiment is regarded as a gold standard for identifying and estimating causal effects in social science. For example, people frequently use survey experiments or conjoint experiments to detect how individual preference for a candidate or immigrant is affected by gender, race, experience, and skills. Researchers' interest extends beyond the causal effects observed within the confines of the experiment; rather, the ultimate aspiration is that these experimental estimates reflect real-world effects. For example, in a survey experiment, if it is observed that voters have a preference for female candidates with an average causal effect of around 2%, it is hoped that, in reality, female candidates indeed garner more votes on average. When such alignment occurs, we describe the experimental effect as being consistent with the real-world effect. This consistency is a fundamental criterion for the relevance of experimental findings to real-world contexts.

In this study, we demonstrate that the gold standard of social science methodology—experiments involving multi-dimensional decision-making processes—often fails to meet the fundamental criterion of consistency. This issue is particularly evident in widely-used social science methodologies such as vignette and conjoint survey experiments, which are crucial for analyzing individual preferences and understanding social behavior. The failure to fulfill basic consistency requirements raises significant doubts about our ability to extrapolate experimental findings to real-world contexts. In particular, we have developed a formal theory to explain why multi-dimensional experiments often fail to meet the three aforementioned consistency requirements. Central to this theory is the concept of attention and salience. While most studies have concentrated on decision theory, our research innovatively applies a behavioral framework to formally investigate these inconsistencies within research methodologies.

We examine three types of consistency within this paper: effect magnitude, effect direction, and attribute importance. 'Consistency of Effect Magnitude' stipulates that the size of the experimental effect should align with that observed in reality. Although this represents a robust criterion, providing the most substantial real-world insight, it is also the most demanding. 'Consistency of Effect Direction' requires that the sign of the experimental effect, whether positive or negative, coincides with the effect direction in the real world. It is apparent that if magnitude consistency is achieved, direction consistency will naturally follow, making it a less stringent requirement, albeit one that yields less real-world detail. Lastly, we address the 'Consistency of Attribute Importance,' which is the need for experimental findings to accurately reflect the relative impact of different treatments or attributes. Researchers often seek to understand which attributes drive the most variation in outcomes, and this form of consistency is also a derivative of magnitude consistency.

Our main finding is that the magnitude of the effect observed in experiments does not align with that in the real world. Specifically, for conjoint experiments, we demonstrate that the experimental effect is significantly exaggerated due to limited attention. Moreover, due to the salience effect, the direction of the experimental effect is often inconsistent with that observed in real-world settings. In essence, while an experimental treatment may exhibit a positive (or negative) effect, the real-world impact could be the opposite. We also prove that the salience effect distorts attribute importance. To support our formal results, we derive several testable hypotheses. We also collect existing empirical evidence and conduct experiments to test those hypotheses. Our studies shed light on the inconsistency problem of experiments from a novel theoretical perspectives.

Authors

• Jiawei Fu, New York University

• Xiaojun Li, University of British Columbia