Judgmental model selection: results and winners!

The “Judgmental model selection for time series forecasting” experiment is now successfully completed! In total, 905 people started the experiment, from which 693 completed the task. Thank you all for your participation in this judgmental experiment. Your contribution to this research is much appreciated.

The experiment itself completed by two approaches:

  • Judgmental model selection, where given 4 unnamed models the participant was asked to select the best for each one of the 32 time series.
  • Judgmental model build, where the participant had to decide on the existence or not of trend and seasonality for each time series. Consequently, the respective forecasting model would be selected.

The user interface of the two approaches is given in Figure 1. Each participant was randomly assigned to either the judgmental model selection or the model build approach.

Judgmental model selection experiments
Fig 1: The two different approaches of this study

As initially advertised, we are going to reward the participants with the best performance, according to their scores. For each participant this was calculated based on the rank of the models corresponding to your choices, according to the actual (out-of-sample) performance of each model. We noticed that there is a statistically significant difference between the two treatments (model selection versus model build), so we will reward the top-10 participants of each approach.

Taking into account any ties in the 10th place of each approach, in total 26 participants will be rewarded with an Amazon e-voucher (£50 each) – total rewards of £1,300 (or $2,200)!

Figure 2 provides the histograms of the scores for the two approaches.

Judgmental model selection winners
Fig. 2: Histograms of the participants’ scores

The winners will be sent their electronic vouchers at the email provided during the registration process of this experiment. The vouchers will be sent out by the end of July 2014.

Later this year, we plan to publish a working paper with the full results of this research. Stay tuned!


Add a Comment