Article by Stone-Romero, Eugene F., and Rosopa, Patrick J. (2008) in Organizational Research Methods, 11:2.
Let say we want to conduct a study with the simplest mediation model, X à M à Y, in order to gain the maximum internal validity inferences, we have to carry out two randomized experiments; one to prove the causal model of X à M and the other one for M à Y. If we choose to conduct quasi-experiments instead of randomized experiments, the internal validity inferences will be reduced; whereas if we choose to conduct non-experimental approach, the causal inferences may not be valid at all (even though we are using ‘causal modeling’ techniques in analyzing the data such as Hierarchical Multiple Regression, Path Analysis and Structural Equation Modeling). This paper outlines the effects of different research design on internal validity inferences:
Research Design |
Control on Confounding Variables |
Internal Validity Inferences |
2 randomized experiments |
Design |
Very Strong |
2 quasi-experiments |
Statistical |
Moderately Strong |
1 randomized experiment |
Design |
Weak |
1 quasi experiment |
Statistical |
Weak |
2 non-experiment |
Statistical |
Very Weak |
1 non-experiment |
Statistical |
Very Weak |
For more complex mediation model, we need to conduct multiple randomized experiments accordingly.
The authors argued that, non-experimental study is only appropriate to prove ‘the consistency between an assumed causal model and the results of the study’, but NOT ‘the consistency between the reality and the results of the study’. That is basically why if non-experimental design approach is selected, we are not legitimately allowed to make any causal inferences. Instead, what we legitimately can claim is the inferences at covariance (e.g. correlation) level only, for example, we can say “…this study has recognized patterns of covariances among measured variables that are consistent with the assumed causal model (specified in the research model)”.
In addition to that (inferences at covariance level), the author argued that we also need to acknowledge that: (a) the same pattern of covariances may also consistent with other causal models, and; (b) our findings do not provide a valid basis for making causal inferences specified in the assumed causal model. For example, we can say (this example is taken directly from this paper) “Hypothesis 1 argued that there would be a positive correlation between X and Y. The test of this hypothesis showed that there was. This finding is consistent with the assumed causal model shown in Figure 1. However, it may also be consistent with a number of other causal models.”
One of the recommendations given by the authors made me smile, I nodded my head few times and I said “yea, you’re right… indeed you’re right!!” Here is the recommendation:
“…we recommend that individuals who teach undergraduate and/or graduate courses in such areas as statistics, research design, research methods, and causal modeling, instruct their students on the inferences that can and can not be made on the basis of data from nonexperimental research. Moreover, they need to disabuse students of the baseless arguments that appear in various publications about the inferences that are appropriate on the basis of ‘causal modeling’ procedures…”