Can we rely on judgment to select the best forecasting model?
Model selection for forecasting problems has attracted much attention during the last 30 years. Many research studies have examined theoretically and empirically different statistical selection methodologies to identify the ‘optimal’ model. The model selection problem is of major practical importance, because if selections are to be done perfectly, substantial gains will achieved in terms of forecasting performance, usually up to 25-30%, and these gains can be translated into profits.
Conventionally, automatic statistical algorithms perform selection of the most appropriate model from a set of alternatives based on the performance of each model or applying a set of rules. Most forecasting packages employ such automatic algorithms, usually introduced as ‘automatic forecasting’, ‘automatic selection’, or ‘expert selection’. Not all of these selection techniques are equally good in terms of design or implementation. Nonetheless, it is easy to devise examples where statistical model selection is pretty disastrous. Such examples are common in real forecasting applications.
Key question: How can you improve your statistical model selection routines to increase your forecasting accuracy?
In addition to the automatic integrated approaches, several forecasting software systems allow the forecasters/demand planners to manually select a method and specify its parameters. Not to mention that this may be the only alternative for many forecasting modules implemented in major ERP systems. So, forecasting support systems offer what we define as ‘judgmental model selection’, meaning that the user (practitioner) can, instead of relying on automatic algorithms, judgmentally select the most appropriate model for each time series.
Key question: How does judgmental model selection perform compared to statistical model selection?
Despite judgmental model selection being an option in many of the leading forecasting systems, there is limited, if any, research with regards to the performance of humans in identifying and ‘optimally’ selecting the best model from a set of alternatives. Moreover, the types of information that are relevant and helpful in judgmental model selection have not been identified.
Key question: What types of information can facilitate the judgmental model selection process?
We try to address these questions by utilizing a large scale judgmental forecasting experiment. If you would like to help us by participating in this research, please follow this link. We are especially interested in participations coming from forecasting practitioners. Moreover, we offer, as a motivation, Amazon vouchers to the participants that achieve the best performance, by selecting the most appropriate models when forecasting.