A Solution to Overfitted Forecasts: An Application to U.S. Governors

• In Event: Political Forecasting Methods: Challenges and Potential Solutions

Abstract

A quadrennial ritual in the study of American politics is the forecasting of U.S. presidential and congressional elections. Though predicting politics is often treated as a side hobby for many scholars, Schrodt (2014) argues, in the tradition of Hempel, that explanation and prediction go together in the production of science. Moreover, forecasting and revision grant scholars the opportunity to “ground truth” of theoretical foundations of political behavior. Theories regarding prospective and retrospective economic voting, the cost of political scandals, and, recently, political polarization all gain dramatically more empirical support when they are used to make accurate out-of-sample predictions, i.e., forecasts.

A common approach to predicting elections relies upon the economic conditions and political situation of the incumbent party. In the election forecasting marketplace of the past fifteen years, model aggregation approaches that utilize the avalanche of horse-race polls released just prior to election day have dominated. However, even horse-race-based models take as a starting point the factors that political science has shown to matter: the fundamentals of political popularity and economic performance (Lewis-Beck 1984; Lewis-Beck & Stegmaier 2014).

Despite the apparent significant predictive power of fundamentals-based forecasts, a central concern with any fundamentals-based forecasts of U.S. elections is the small number of elections for model training. For the most common type of forecast, presidential elections, there are no more than 18 post-war elections for training and many more potential predictors. Focusing on governors’ races dramatically increases the number of elections, but data availability, and the potential complexity of a sub-national fundamentals model, produces its own challenges. Overall, the small size of the training data and the myriads of potential predictors mean that many fundamentals-based forecasts are at substantial risk of overfitting.

This paper presents a potential solution to the challenges of overfitting in gubernatorial election forecasting (and, by extension, presidential races). First, our forecast for U.S. governors overcomes the issue of limited gubernatorial approval data by utilizing a new dataset of governors’ approval data that covers 48 states going back, in some cases, to the 1980s (Singer 2023). This dataset also contains state-level presidential approval, which we combine with economic measures from the St. Louis Fed and other political factors to produce a robust set of predictors.

The second challenge is the selection of predictors. With a relatively small training dataset, including all theoretically related predictors creates conditions for model overfitting (Klarner 2012, includes 29 predictors). To create a parsimonious model, we turn to recent advances in machine learning by utilizing a LASSO algorithm to select predictors from a theoretically relevant set of options. In a wide range of applications, from clinical health decisions to economics, LASSO has been shown to produce a set of predictors to produce a “good” model fit without overfitting.

We validate our forecast, based on LASSO selected governor approval, presidential approval, past presidential result, and partisanship, via two out-of-sample approaches: k-folds validation and predicting the 2022 governors' races from a training set that ended with 2020. Overall, our model shows substantial predictive power. Our root mean squared error (RMSE) is substantially smaller than a standard expert prediction (CPR PVI). With the removal of three notable a priori outlier cases of 2022 (MA, MD, and HI), the RMSE of our model based on factors measured >6 months before the election is identical to 548’s election-day horse-race-based forecast.

Authors

• Ryan Carlin, Georgia State University

• Gregory Love, University of Mississippi

• Matthew Singer, University of Connecticut