Are IMF Programs Effective? Assessing External Validity of Regression Analysis

• In Event: IMF and Its Effectiveness

Abstract

In the past decade, scholars of Political Science in general and International Political Economy (IPE) in particular have increasingly resorted to experimental designs in order to test their hypotheses. Experimental designs create an artificial lab-like environment for testing hypotheses, which helps identify causal effects, but they are criticized for suffering from lower external validity compared with big-data econometric models that are based on observational data. In their defense, experimentalists retort that inference from regression analysis of observational data often relies on far fewer data points than implied by the entire dataset used. Experimental designs are therefore preferable to observational studies on internal validity grounds and are no worse in terms of the generalizability of the findings. No real tradeoff exists between internal and external validity in the choice between these two types of research designs.
We examine whether this is true. We develop several indicators of effective samples in econometric models that tell us if a particular estimate is based on the entire data fed into the regression, or rather on a narrower effective subset of observations (implying lower external validity). These indicators are easy to produce, intuitive to understand, and computable for a range of models, including linear, multi-step or selection models, generalized linear models estimates (logit, probit, multinomial logit or probit, Poisson regression, and parametric or semiparametric duration models) and random coefficient models.
The International Monetary Fund (IMF) has attracted abundant research on its policies and practices. The Fund’s mandate is to avert financial crises and promote economic growth. However, according to the literature on the effectiveness of IMF programs and interventions, the track-record on both outcomes is mixed. Some studies find that IMF programs harm economic growth, others contend that this effect is due to adverse selection. IMF programs encourage current account liberalization and enhance credit ratings, but this may increase moral hazard and vulnerability to the volatility of transnational financial flows. IMF programs only partly catalyse private and bilateral foreign aid and direct investments. Conditionality catalyses more FDI, but may be counterproductive in some cases. More broadly, the evidence suggests that IMF programs may increase poverty and income inequality, and have adverse and even gendered effects on unemployment, and labour income and rights. IMF programs are also linked to deteriorating public health, educational outcomes, vaccination rates, child mortality and corruption. There is some evidence that IMF programs also increase government instability and the likelihood of civil war.
This study extends the findings of the IMF effectiveness literature in a critical dimension. Over the past decades, researchers have been vexed by the challenge of identifying the treatment effect of IMF programs. The focus on internal validity has relegated external validity to a neglected second-order issue. Consequently, our knowledge about to what extent the findings from this research program generalize across different countries and different time periods has remained extremely limited. We begin to fill this gap. We review current literature on the effectiveness of IMF interventions, replicate almost 70 recent relevant studies that were published in top peer-reviewed journals, and run our diagnostics on them. We uncover huge variation in effective sample size for specific IMF treatments. These can occur within the same publication and even within the same model. In general, external validity in this field is relatively poor. About half of the IMF program treatment estimates effectively rely on fewer than one-quarter of the observations. No study had an effective sample over 70 percent of the nominal sample. In one study recently published in one of the most prominent journals in the field, the headline result is effectively driven by the particular cases of one or two countries, or a single time period. These findings cast some doubt about the external validity of the published findings.
We also examine the determinants of effective sample size. We find that models using an IMF program dummy generally benefit from enhanced external validity. Adding control variables and fixed effects reduces external validity, while having multiple treatments in a single model does not. Importantly, instrumental-variable regression significantly decreases the relative effective sample. Thus, in contrast to experimentalists’ arguments we find that researchers do face a critical trade-off: the promise of enhanced internal validity comes at the cost of external validity. The indicators of external validity that we develop can help scholars manage and optimize this trade-off. In the concluding section we suggest some dos and don’ts to help scholars optimize their research designs.

Authors

• Tal Sadeh, Tel Aviv University

• Bernhard Reinsberg, University of Glasgow

• Gal Bitton, Tel Aviv University