The “Judgmental model selection for time series forecasting” experiment is now successfully completed! In total, 905 people started the experiment, from which 693 completed the task. Thank you all for your participation in this judgmental experiment. Your contribution to this research is much appreciated.
The experiment itself completed by two approaches:
Judgmental model selection, where given 4 unnamed models the participant was asked to select the best for each one of the 32 time series.
Judgmental model build, where the participant had to decide on the existence or not of trend and seasonality for each time series. Consequently, the respective forecasting model would be selected.
The user interface of the two approaches is given in Figure 1. Each participant was randomly assigned to either the judgmental model selection or the model build approach.
As initially advertised, we are going to reward the participants with the best performance, according to their scores. For each participant this was calculated based on the rank of the models corresponding to your choices, according to the actual (out-of-sample) performance of each model. We noticed that there is a statistically significant difference between the two treatments (model selection versus model build), so we will reward the top-10 participants of each approach.
Taking into account any ties in the 10th place of each approach, in total 26 participants will be rewarded with an Amazon e-voucher (£50 each) – total rewards of £1,300 (or $2,200)!
Figure 2 provides the histograms of the scores for the two approaches.
The winners will be sent their electronic vouchers at the email provided during the registration process of this experiment. The vouchers will be sent out by the end of July 2014.
Later this year, we plan to publish a working paper with the full results of this research. Stay tuned!
Dr Fotios Petropoulos is Lecturer (Assistant Professor) at Cardiff Business School of Cardiff University. He remained the Coordinator and Research Associate of the Forecasting & Strategy Unit of NTUA for nine years, where he also completed his Doctor in Engineering. After that, he acted as a Senior Research Associate at the Lancaster Centre for Forecasting of Lancaster University. Fotios is engaged in time series forecasting, judgmental approaches for forecasting, optimal model selection and business forecasting processes. He has provided knowledge transfer and consultancy in Retail, Supply Chain, Spare Parts and Utilities. Except from that, he is an expert in software analysis, system design and development of integrated forecasting support systems. He has published several original articles in high-ranked academic journals and presented his work in major international conferences. Fotios is the founder of the Forecasting Society.