Benchmarking

You can easily compare predictive performance of multiple algorithms such as Silverkite and Prophet using the BenchmarkForecastConfig class. In this tutorial we describe the step-by-step process of defining, running and monitoring a benchmark. We also demonstrate how to use the class functions to compute and plot errors for multiple models.

12 from dataclasses import replace
13
14 import plotly
15 import plotly.graph_objects as go
16
17 from greykite.common.evaluation import EvaluationMetricEnum
18 from greykite.framework.benchmark.benchmark_class import BenchmarkForecastConfig
19 from greykite.framework.benchmark.data_loader_ts import DataLoaderTS
20 from greykite.framework.templates.autogen.forecast_config import ComputationParam
21 from greykite.framework.templates.autogen.forecast_config import EvaluationMetricParam
22 from greykite.framework.templates.autogen.forecast_config import EvaluationPeriodParam
23 from greykite.framework.templates.autogen.forecast_config import MetadataParam
24 from greykite.framework.templates.autogen.forecast_config import ForecastConfig
25 from greykite.framework.templates.autogen.forecast_config import ModelComponentsParam
26 from greykite.framework.templates.model_templates import ModelTemplateEnum
27 from greykite.sklearn.cross_validation import RollingTimeSeriesSplit

Load the data

First load your dataset into a pandas dataframe. We will use the peyton-manning dataset as a running example.

35 # Loads dataset into UnivariateTimeSeries
36 dl = DataLoaderTS()
37 ts = dl.load_peyton_manning_ts()
38 df = ts.df  # cleaned pandas.DataFrame

Define the Configs

We specify the models we want to benchmark via the configs parameter. In this example we will benchmark 1 Prophet and 2 different Silverkite models. We first define the common components of the models such as MetadataParam and EvaluationMetricParam, and then update the configuration to specify individual models.

49 ## Define common components  of the configs
50 # Specifies dataset information
51 metadata = MetadataParam(
52     time_col="ts",   # name of the time column
53     value_col="y",   # name of the value column
54     freq="D"         # "H" for hourly, "D" for daily, "W" for weekly, etc.
55 )
56
57 # Defines number of periods to forecast into the future
58 forecast_horizon = 7
59
60 # Specifies intended coverage of the prediction interval
61 coverage = 0.95
62
63 # Defines the metrics to evaluate the forecasts
64 # We use Mean Absolute Percent Error (MAPE) in this tutorial
65 evaluation_metric = EvaluationMetricParam(
66     cv_selection_metric=EvaluationMetricEnum.MeanAbsolutePercentError.name,
67     cv_report_metrics=None
68 )
69
70 # Defines the cross-validation config within pipeline
71 evaluation_period = EvaluationPeriodParam(
72     cv_max_splits=1,  # Benchmarking n_splits is defined in tscv, here we don't need split to choose parameter sets
73     periods_between_train_test=0,
74 )
75
76 # Defines parameters related to grid-search computation
77 computation = ComputationParam(
78     hyperparameter_budget=None,
79     n_jobs=-1,  # to debug, change to 1 for more informative error messages
80     verbose=3)
81
82 # Defines common components across all the configs
83 # ``model_template`` and ``model_components_param`` changes between configs
84 common_config = ForecastConfig(
85     metadata_param=metadata,
86     forecast_horizon=forecast_horizon,
87     coverage=coverage,
88     evaluation_metric_param=evaluation_metric,
89     evaluation_period_param=evaluation_period,
90     computation_param=computation,
91 )

Now we update common_config to specify the individual models.

 96 # # The following code defines a ``Prophet`` configuration.
 97 # # Defines ``Prophet`` model template with custom seasonality
 98 # model_components = ModelComponentsParam(
 99 #     seasonality={
100 #             "seasonality_mode": ["additive"],
101 #             "yearly_seasonality": ["auto"],
102 #             "weekly_seasonality": [True],
103 #         },
104 #         growth={
105 #             "growth_term": ["linear"]
106 #         }
107 # )
108 # param_update = dict(
109 #     model_template=ModelTemplateEnum.PROPHET.name,
110 #     model_components_param=model_components
111 # )
112 # Prophet = replace(common_config, **param_update)
113
114 # Defines ``Silverkite`` model template with automatic autoregression
115 # and changepoint detection
116 model_components = ModelComponentsParam(
117     changepoints={
118         "changepoints_dict": {
119             "method": "auto",
120         }
121     },
122     autoregression={
123         "autoreg_dict": "auto"
124     }
125 )
126 param_update = dict(
127     model_template=ModelTemplateEnum.SILVERKITE.name,
128     model_components_param=model_components
129 )
130 Silverkite_1 = replace(common_config, **param_update)
131
132 # Defines ``Silverkite`` model template via string encoding
133 param_update = dict(
134     model_template="DAILY_SEAS_NMQM_GR_LINEAR_CP_NM_HOL_SP2_FEASET_AUTO_ALGO_RIDGE_AR_AUTO_DSI_AUTO_WSI_AUTO",
135     model_components_param=None
136 )
137 Silverkite_2 = replace(common_config, **param_update)
138
139 # Define the list of configs to benchmark
140 # The dictionary keys will be used to store the benchmark results
141 configs = {
142     # "Prophet": Prophet,
143     "SK_1": Silverkite_1,
144     "SK_2": Silverkite_2,
145 }

Define the Cross-Validation (CV)

In time-series forecasting we use a Rolling Window CV. You can easily define it by using RollingTimeSeriesSplit class. The CV parameters depend on the data frequency, forecast horizon as well as the speed of the models. See Benchmarking documentation for guidance on how to choose CV parameters for your use case.

158 # Define the benchmark folds
159 # CV parameters are changed for illustration purpose
160 tscv = RollingTimeSeriesSplit(
161     forecast_horizon=forecast_horizon,
162     min_train_periods=2 * 365,
163     expanding_window=True,
164     use_most_recent_splits=True,
165     periods_between_splits=5,
166     periods_between_train_test=0,
167     max_splits=4)  # reduced to 4 from 16 for faster runtime
168
169 # Print the train, test split for benchmark folds
170 for split_num, (train, test) in enumerate(tscv.split(X=df)):
171     print(split_num, train, test)

Out:

0 [   0    1    2 ... 2939 2940 2941] [2942 2943 2944 2945 2946 2947 2948]
1 [   0    1    2 ... 2944 2945 2946] [2947 2948 2949 2950 2951 2952 2953]
2 [   0    1    2 ... 2949 2950 2951] [2952 2953 2954 2955 2956 2957 2958]
3 [   0    1    2 ... 2954 2955 2956] [2957 2958 2959 2960 2961 2962 2963]

Run the Benchmark

To start the benchmarking procedure execute its run method.

If you get an error message at this point, then there is a compatibility issue between your benchmark inputs. Check Debugging the Benchmark section for instructions on how to derive valid inputs.

181 bm = BenchmarkForecastConfig(df=df, configs=configs, tscv=tscv)
182 bm.run()

Out:

  0%|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | 0/2 [00:00<?, ?it/s]
Benchmarking 'SK_1' :   0%|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | 0/2 [00:00<?, ?it/s]

  0%|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | 0/4 [00:00<?, ?it/s]

Split '0' :   0%|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | 0/4 [00:00<?, ?it/s]Fitting 1 folds for each of 1 candidates, totalling 1 fits


Split '0' :  25%|############################################################################################################################################################################################2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | 1/4 [00:40<02:02, 40.80s/it]

Split '1' :  25%|############################################################################################################################################################################################2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | 1/4 [00:40<02:02, 40.80s/it]Fitting 1 folds for each of 1 candidates, totalling 1 fits


Split '1' :  50%|########################################################################################################################################################################################################################################################################################################################################################################################5                                                                                                                                                                                                                                                                                                                                                                                        | 2/4 [01:12<01:10, 35.41s/it]

Split '2' :  50%|########################################################################################################################################################################################################################################################################################################################################################################################5                                                                                                                                                                                                                                                                                                                                                                                        | 2/4 [01:12<01:10, 35.41s/it]Fitting 1 folds for each of 1 candidates, totalling 1 fits


Split '2' :  75%|####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################7                                                                                                                                                                                            | 3/4 [01:46<00:35, 35.01s/it]

Split '3' :  75%|####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################7                                                                                                                                                                                            | 3/4 [01:46<00:35, 35.01s/it]Fitting 1 folds for each of 1 candidates, totalling 1 fits


Split '3' : 100%|#################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################| 4/4 [02:21<00:00, 34.94s/it]
Split '3' : 100%|#################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################| 4/4 [02:21<00:00, 35.45s/it]

Benchmarking 'SK_1' :  50%|###################################################################################################################################################################################################################################################################################################################################################################################                                                                                                                                                                                                                                                                                                                                                                                   | 1/2 [02:21<02:21, 141.81s/it]
Benchmarking 'SK_2' :  50%|###################################################################################################################################################################################################################################################################################################################################################################################                                                                                                                                                                                                                                                                                                                                                                                   | 1/2 [02:21<02:21, 141.81s/it]

  0%|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | 0/4 [00:00<?, ?it/s]

Split '0' :   0%|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | 0/4 [00:00<?, ?it/s]Fitting 1 folds for each of 1 candidates, totalling 1 fits


Split '0' :  25%|############################################################################################################################################################################################2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | 1/4 [00:29<01:28, 29.56s/it]

Split '1' :  25%|############################################################################################################################################################################################2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | 1/4 [00:29<01:28, 29.56s/it]Fitting 1 folds for each of 1 candidates, totalling 1 fits


Split '1' :  50%|########################################################################################################################################################################################################################################################################################################################################################################################5                                                                                                                                                                                                                                                                                                                                                                                        | 2/4 [00:58<00:58, 29.18s/it]

Split '2' :  50%|########################################################################################################################################################################################################################################################################################################################################################################################5                                                                                                                                                                                                                                                                                                                                                                                        | 2/4 [00:58<00:58, 29.18s/it]Fitting 1 folds for each of 1 candidates, totalling 1 fits


Split '2' :  75%|####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################7                                                                                                                                                                                            | 3/4 [01:28<00:29, 29.51s/it]

Split '3' :  75%|####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################7                                                                                                                                                                                            | 3/4 [01:28<00:29, 29.51s/it]Fitting 1 folds for each of 1 candidates, totalling 1 fits


Split '3' : 100%|#################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################| 4/4 [01:56<00:00, 28.91s/it]
Split '3' : 100%|#################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################| 4/4 [01:56<00:00, 29.09s/it]

Benchmarking 'SK_2' : 100%|######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################| 2/2 [04:18<00:00, 126.85s/it]
Benchmarking 'SK_2' : 100%|######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################| 2/2 [04:18<00:00, 129.09s/it]

Monitor the Benchmark

During benchmarking a couple of color coded progress bars are displayed to inform the user of the advancement of the entire process. The first bar displays config level information, while the second bar displays split level information for the current config. See example in Benchmarking documentation.

On the left side of the progress bar, it shows which config/ split is currently being benchmarked and progress within that level as a percentage.

On the right side, the user can see how many configs/ splits have been benchmarked and how many are remaining. Additionally, this bar also displays elapsed time and remaining runtime for the corresponding level.

Benchmark Output

The output of a successful benchmark procedure is stored as a nested dictionary under the class attribute result. For details on the structure of this tree check Benchmarking documentation.

You can extract any specific information by navigating this tree. For example, you can check the summary and component plot of any config.

209 # Check summary of SK_1 model on first fold
210 model = bm.result["SK_2"]["rolling_evaluation"]["split_0"]["pipeline_result"].model
211 model[-1].summary(max_colwidth=30)

Out:

============================ Forecast Model Summary ============================

Number of observations: 2942,   Number of features: 264
Method: Ridge regression
Number of nonzero features: 264
Regularization parameter: 3.29

Residuals:
         Min           1Q       Median           3Q          Max
      -2.261      -0.2428      -0.0543       0.1658         3.23

                     Pred_col  Estimate Std. Err Pr(>)_boot sig. code                 95%CI
                    Intercept     6.117   0.1325     <2e-16       ***        (5.873, 6.399)
      events_Chinese New Year   0.04014   0.1923      0.844               (-0.3123, 0.4445)
    events_Chinese New Year-1   -0.1147    0.164      0.478                (-0.4074, 0.222)
    events_Chinese New Year-2   0.09564   0.1636      0.560                (-0.207, 0.4024)
    events_Chinese New Year+1   0.06808   0.1787      0.710                (-0.275, 0.4365)
    events_Chinese New Year+2    0.1144    0.132      0.414                (-0.132, 0.3835)
         events_Christmas Day   -0.3427   0.1239      0.006        **     (-0.602, -0.1265)
       events_Christmas Day-1   -0.1638   0.1342      0.202              (-0.4494, 0.08585)
       events_Christmas Day-2  -0.02716   0.1769      0.886                (-0.395, 0.2888)
       events_Christmas Day+1   -0.2164   0.1308      0.094         .    (-0.4949, 0.01968)
       events_Christmas Day+2    0.1633  0.09634      0.090         .    (-0.02193, 0.3574)
events_Easter...hern Ireland]   -0.1713  0.08193      0.040         *   (-0.3241, 0.002696)
events_Easter...rn Ireland]-1   -0.1137  0.04745      0.018         *   (-0.2013, -0.01816)
events_Easter...rn Ireland]-2  -0.07558  0.04327      0.082         .    (-0.1602, 0.01076)
events_Easter...rn Ireland]+1   -0.1081  0.05922      0.068         .   (-0.2232, 0.008614)
events_Easter...rn Ireland]+2  -0.03451  0.04798      0.456              (-0.1276, 0.06185)
           events_Good Friday   -0.1708  0.05684      0.002        **   (-0.2682, -0.04581)
         events_Good Friday-1   -0.1334  0.05417      0.020         *  (-0.2241, -0.006718)
         events_Good Friday-2  -0.07375  0.05927      0.230              (-0.1883, 0.03024)
         events_Good Friday+1  -0.07558  0.04327      0.082         .    (-0.1602, 0.01076)
         events_Good Friday+2   -0.1137  0.04745      0.018         *   (-0.2013, -0.01816)
      events_Independence Day   0.00592  0.05686      0.920               (-0.1051, 0.1168)
    events_Independence Day-1  -0.02185  0.05398      0.692              (-0.1323, 0.08326)
    events_Independence Day-2  -0.04206  0.04725      0.370              (-0.1365, 0.05346)
    events_Independence Day+1  -0.06111  0.05764      0.280              (-0.1625, 0.05779)
    events_Independence Day+2  -0.06432   0.0609      0.278              (-0.1822, 0.05475)
             events_Labor Day   -0.2295  0.07977      0.004        **    (-0.3699, -0.0639)
           events_Labor Day-1  -0.09774  0.07572      0.200              (-0.2489, 0.05531)
           events_Labor Day-2  -0.03347  0.06103      0.584              (-0.1545, 0.08419)
           events_Labor Day+1   -0.1579  0.06141      0.010         *   (-0.2757, -0.03354)
           events_Labor Day+2   -0.1835  0.05204     <2e-16       ***   (-0.2754, -0.07569)
          events_Memorial Day   -0.3278   0.0581     <2e-16       ***    (-0.4284, -0.2022)
        events_Memorial Day-1   -0.2063  0.06271      0.002        **   (-0.3214, -0.07642)
        events_Memorial Day-2  -0.04419  0.07312      0.538                (-0.179, 0.1071)
        events_Memorial Day+1   -0.1079  0.04415      0.018         *   (-0.1887, -0.02103)
        events_Memorial Day+2   0.08969  0.06883      0.202              (-0.03874, 0.2262)
         events_New Years Day   0.01326  0.08453      0.856               (-0.1435, 0.1766)
       events_New Years Day-1    0.1042   0.1044      0.322              (-0.09111, 0.2991)
       events_New Years Day-2    0.2211   0.1345      0.106              (-0.05025, 0.4778)
       events_New Years Day+1     0.266   0.1104      0.006        **     (0.05909, 0.4818)
       events_New Years Day+2    0.2402   0.1247      0.050         .   (-0.006849, 0.4703)
                 events_Other   0.01332  0.03153      0.708               (-0.044, 0.07875)
               events_Other-1   0.01768  0.03187      0.578             (-0.04095, 0.08127)
               events_Other-2   0.03747  0.02806      0.182             (-0.01706, 0.08939)
               events_Other+1   0.03224  0.02927      0.278             (-0.02433, 0.08444)
               events_Other+2    0.0264  0.02719      0.332             (-0.02755, 0.08045)
          events_Thanksgiving   -0.2122  0.06654     <2e-16       ***     (-0.338, -0.0773)
        events_Thanksgiving-1   -0.3417  0.06408     <2e-16       ***    (-0.4565, -0.1992)
        events_Thanksgiving-2   -0.2734  0.06771     <2e-16       ***    (-0.3958, -0.1131)
        events_Thanksgiving+1    -0.142  0.07908      0.060         .    (-0.2741, 0.02046)
        events_Thanksgiving+2   -0.1877  0.05847     <2e-16       ***   (-0.3062, -0.07507)
          events_Veterans Day  -0.02697  0.08165      0.754               (-0.1756, 0.1392)
        events_Veterans Day-1  -0.05888  0.07079      0.432              (-0.1931, 0.06346)
        events_Veterans Day-2  -0.03315  0.08606      0.720                (-0.204, 0.1394)
        events_Veterans Day+1  -0.02047  0.07337      0.784               (-0.1535, 0.1359)
        events_Veterans Day+2  -0.02765  0.06597      0.650               (-0.1545, 0.1107)
                str_dow_2-Tue   0.01169  0.01985      0.534             (-0.02727, 0.05043)
                str_dow_3-Wed  -0.02174   0.0169      0.196            (-0.05765, 0.009198)
                str_dow_4-Thu   -0.0307   0.0172      0.078         .  (-0.06299, 0.005312)
                str_dow_5-Fri  -0.03898  0.01669      0.016         * (-0.07364, -0.009126)
                str_dow_6-Sat   -0.1091  0.01849     <2e-16       ***   (-0.1475, -0.07406)
                str_dow_7-Sun   0.01308  0.02013      0.520              (-0.0244, 0.05262)
                          ct1  0.005591   0.0779      0.932               (-0.1433, 0.1688)
               is_weekend:ct1 -0.005726  0.04574      0.898             (-0.08998, 0.08496)
            str_dow_2-Tue:ct1   0.01478  0.04739      0.752              (-0.08184, 0.1066)
            str_dow_3-Wed:ct1  -0.01108  0.03536      0.782             (-0.07699, 0.05619)
            str_dow_4-Thu:ct1  -0.01126   0.0371      0.782              (-0.08185, 0.0615)
            str_dow_5-Fri:ct1   0.03008  0.03494      0.388              (-0.03929, 0.0997)
            str_dow_6-Sat:ct1   0.01223  0.03466      0.718             (-0.05361, 0.07797)
            str_dow_7-Sun:ct1  -0.01795  0.04612      0.702              (-0.1046, 0.07199)
            cp0_2008_09_29_00    0.2107  0.02978     <2e-16       ***      (0.1496, 0.2633)
 is_weekend:cp0_2008_09_29_00   0.04814   0.0195      0.008        **    (0.01395, 0.08829)
str_dow_2-Tue...2008_09_29_00     0.033  0.02576      0.210             (-0.01772, 0.08353)
str_dow_3-Wed...2008_09_29_00   0.01978  0.01808      0.274              (-0.0157, 0.05331)
str_dow_4-Thu...2008_09_29_00   0.01765  0.01966      0.374             (-0.01994, 0.05588)
str_dow_5-Fri...2008_09_29_00   0.03991  0.02163      0.066         .  (-0.005792, 0.08154)
str_dow_6-Sat...2008_09_29_00   0.02829  0.02025      0.170             (-0.01334, 0.06648)
str_dow_7-Sun...2008_09_29_00   0.01987  0.02269      0.402             (-0.02131, 0.06487)
            cp1_2008_11_03_00    0.2332  0.02882     <2e-16       ***       (0.171, 0.2803)
 is_weekend:cp1_2008_11_03_00   0.05369  0.01875      0.006        **    (0.01953, 0.09501)
str_dow_2-Tue...2008_11_03_00   0.03584   0.0239      0.134             (-0.01221, 0.08212)
str_dow_3-Wed...2008_11_03_00     0.025  0.01738      0.144             (-0.00855, 0.05765)
str_dow_4-Thu...2008_11_03_00   0.02347  0.01791      0.202            (-0.009179, 0.05937)
str_dow_5-Fri...2008_11_03_00   0.04009  0.02034      0.062         .  (-0.001562, 0.08069)
str_dow_6-Sat...2008_11_03_00   0.03256  0.01912      0.088         .  (-0.004081, 0.06813)
str_dow_7-Sun...2008_11_03_00   0.02114  0.02205      0.362             (-0.01697, 0.06522)
            cp2_2008_12_15_00    0.2379  0.03133     <2e-16       ***      (0.1708, 0.2902)
 is_weekend:cp2_2008_12_15_00    0.0537   0.0204      0.006        **    (0.01579, 0.09664)
str_dow_2-Tue...2008_12_15_00   0.03843  0.02234      0.090         .  (-0.006632, 0.08177)
str_dow_3-Wed...2008_12_15_00    0.0245  0.01732      0.168             (-0.009413, 0.0574)
str_dow_4-Thu...2008_12_15_00   0.02529    0.017      0.138            (-0.008289, 0.05976)
str_dow_5-Fri...2008_12_15_00   0.03431  0.02005      0.098         .  (-0.006721, 0.07598)
str_dow_6-Sat...2008_12_15_00   0.03146  0.01889      0.076         .  (-0.005801, 0.06696)
str_dow_7-Sun...2008_12_15_00   0.02225  0.02319      0.366             (-0.02104, 0.06678)
            cp3_2009_01_12_00    0.2351  0.03468     <2e-16       ***      (0.1623, 0.2956)
 is_weekend:cp3_2009_01_12_00   0.05406  0.02223      0.014         *    (0.01222, 0.09988)
str_dow_2-Tue...2009_01_12_00   0.03763  0.02232      0.094         .  (-0.008708, 0.07972)
str_dow_3-Wed...2009_01_12_00   0.02275  0.01811      0.210             (-0.01189, 0.05845)
str_dow_4-Thu...2009_01_12_00   0.02356  0.01739      0.174             (-0.009737, 0.0583)
str_dow_5-Fri...2009_01_12_00    0.0306  0.02004      0.140            (-0.006915, 0.07263)
str_dow_6-Sat...2009_01_12_00   0.02925  0.01938      0.112            (-0.007534, 0.06567)
str_dow_7-Sun...2009_01_12_00   0.02482  0.02434      0.324             (-0.02227, 0.07111)
            cp4_2009_12_21_00   -0.1748  0.06316      0.002        **   (-0.2945, -0.05474)
 is_weekend:cp4_2009_12_21_00  -0.06254  0.04681      0.202              (-0.1528, 0.02638)
str_dow_2-Tue...2009_12_21_00  -0.05231  0.04538      0.250               (-0.1369, 0.0459)
str_dow_3-Wed...2009_12_21_00  -0.01102  0.03577      0.786             (-0.08066, 0.06068)
str_dow_4-Thu...2009_12_21_00  -0.00327  0.02965      0.916             (-0.05568, 0.05794)
str_dow_5-Fri...2009_12_21_00  -0.05109  0.04022      0.186               (-0.1382, 0.0258)
str_dow_6-Sat...2009_12_21_00  -0.03847  0.03684      0.302              (-0.1031, 0.03347)
str_dow_7-Sun...2009_12_21_00  -0.02408  0.04782      0.636              (-0.1211, 0.05885)
            cp5_2010_01_25_00   -0.1655  0.06737      0.016         *   (-0.3032, -0.03955)
 is_weekend:cp5_2010_01_25_00  -0.05777  0.05127      0.288              (-0.1526, 0.03681)
str_dow_2-Tue...2010_01_25_00  -0.05328  0.04945      0.284              (-0.1457, 0.05164)
str_dow_3-Wed...2010_01_25_00  -0.01018   0.0384      0.800             (-0.08645, 0.06589)
str_dow_4-Thu...2010_01_25_00 -0.003794  0.03091      0.916             (-0.05938, 0.06199)
str_dow_5-Fri...2010_01_25_00  -0.05383   0.0416      0.174              (-0.1411, 0.02637)
str_dow_6-Sat...2010_01_25_00  -0.04081  0.03963      0.312               (-0.107, 0.03639)
str_dow_7-Sun...2010_01_25_00  -0.01698  0.05263      0.764              (-0.1232, 0.07659)
            cp6_2011_02_14_00    0.2477   0.0857      0.002        **        (0.07327, 0.4)
 is_weekend:cp6_2011_02_14_00   0.08774  0.06294      0.168              (-0.03296, 0.2063)
str_dow_2-Tue...2011_02_14_00 -0.002566  0.06113      0.968               (-0.1282, 0.1184)
str_dow_3-Wed...2011_02_14_00   0.04493   0.0482      0.368                (-0.05029, 0.13)
str_dow_4-Thu...2011_02_14_00   0.03484  0.04569      0.458               (-0.05847, 0.116)
str_dow_5-Fri...2011_02_14_00   0.01418  0.05573      0.814              (-0.08938, 0.1227)
str_dow_6-Sat...2011_02_14_00   0.02888  0.05221      0.606              (-0.07403, 0.1221)
str_dow_7-Sun...2011_02_14_00   0.05888  0.06006      0.342              (-0.06005, 0.1622)
            cp7_2012_02_27_00   -0.4776  0.07535     <2e-16       ***     (-0.622, -0.3163)
 is_weekend:cp7_2012_02_27_00   -0.1035   0.0508      0.052         .   (-0.1997, 0.005238)
str_dow_2-Tue...2012_02_27_00  -0.08778  0.05949      0.142               (-0.209, 0.01926)
str_dow_3-Wed...2012_02_27_00  -0.07799  0.06562      0.240              (-0.2185, 0.03687)
str_dow_4-Thu...2012_02_27_00  -0.07495   0.0566      0.186              (-0.2006, 0.02974)
str_dow_5-Fri...2012_02_27_00  -0.04902  0.05879      0.422              (-0.1648, 0.06958)
str_dow_6-Sat...2012_02_27_00  -0.04071  0.04959      0.434               (-0.146, 0.04975)
str_dow_7-Sun...2012_02_27_00  -0.06282  0.05546      0.254              (-0.1647, 0.05075)
            cp8_2012_04_09_00   -0.4308  0.06356     <2e-16       ***    (-0.5446, -0.2965)
 is_weekend:cp8_2012_04_09_00   -0.1019  0.04733      0.032         *  (-0.1901, -0.006604)
str_dow_2-Tue...2012_04_09_00  -0.07723  0.05902      0.192               (-0.2003, 0.0329)
str_dow_3-Wed...2012_04_09_00  -0.06449   0.0623      0.310              (-0.1925, 0.04622)
str_dow_4-Thu...2012_04_09_00  -0.06451  0.05378      0.230              (-0.1757, 0.04013)
str_dow_5-Fri...2012_04_09_00  -0.05378  0.05577      0.332                (-0.165, 0.0535)
str_dow_6-Sat...2012_04_09_00   -0.0419  0.04741      0.356              (-0.1416, 0.04353)
str_dow_7-Sun...2012_04_09_00  -0.06008  0.05259      0.264               (-0.1628, 0.0399)
            cp9_2013_03_04_00    0.2743  0.08911     <2e-16       ***     (0.08415, 0.4244)
 is_weekend:cp9_2013_03_04_00  0.004187  0.06212      0.954               (-0.1118, 0.1171)
str_dow_2-Tue...2013_03_04_00   0.08094  0.05987      0.156              (-0.04387, 0.1922)
str_dow_3-Wed...2013_03_04_00   0.07103  0.04193      0.094         .    (-0.01243, 0.1586)
str_dow_4-Thu...2013_03_04_00   0.06013  0.04796      0.214              (-0.03709, 0.1541)
str_dow_5-Fri...2013_03_04_00   0.03473  0.05272      0.536              (-0.07052, 0.1301)
str_dow_6-Sat...2013_03_04_00  0.008695  0.04945      0.846              (-0.08724, 0.1028)
str_dow_7-Sun...2013_03_04_00 -0.004499  0.06793      0.960               (-0.1381, 0.1163)
           cp10_2014_01_27_00   -0.2157  0.09681      0.024         *   (-0.3854, -0.03092)
is_weekend:cp10_2014_01_27_00  -0.05859  0.06633      0.408              (-0.1966, 0.05745)
str_dow_2-Tue...2014_01_27_00   0.04794   0.1101      0.656               (-0.1592, 0.2724)
str_dow_3-Wed...2014_01_27_00  -0.05173  0.06488      0.420              (-0.1668, 0.07595)
str_dow_4-Thu...2014_01_27_00  -0.03651  0.06754      0.596              (-0.1779, 0.08798)
str_dow_5-Fri...2014_01_27_00  -0.01177  0.07505      0.910               (-0.1642, 0.1148)
str_dow_6-Sat...2014_01_27_00  -0.05403   0.0597      0.358              (-0.1673, 0.06688)
str_dow_7-Sun...2014_01_27_00  -0.00464  0.08801      0.974                (-0.193, 0.1402)
          ct1:sin1_tow_weekly  -0.00771  0.03471      0.826             (-0.07497, 0.06088)
          ct1:cos1_tow_weekly    -0.016  0.05914      0.786                (-0.13, 0.09561)
          ct1:sin2_tow_weekly    0.0381  0.03822      0.314              (-0.03618, 0.1146)
          ct1:cos2_tow_weekly 8.192e-05  0.05565      1.000               (-0.1204, 0.1008)
cp0_2008_09_2...n1_tow_weekly -0.003941   0.0166      0.810              (-0.03712, 0.0297)
cp0_2008_09_2...s1_tow_weekly   0.01203  0.02856      0.654             (-0.04876, 0.06494)
cp0_2008_09_2...n2_tow_weekly   0.01739  0.01873      0.342             (-0.01911, 0.05578)
cp0_2008_09_2...s2_tow_weekly   0.01747  0.02746      0.498             (-0.04275, 0.07236)
cp1_2008_11_0...n1_tow_weekly -0.001587  0.01568      0.924             (-0.03233, 0.02909)
cp1_2008_11_0...s1_tow_weekly   0.01101  0.02562      0.634              (-0.04805, 0.0584)
cp1_2008_11_0...n2_tow_weekly   0.01571  0.01747      0.368             (-0.01927, 0.05115)
cp1_2008_11_0...s2_tow_weekly   0.01604  0.02478      0.490             (-0.03528, 0.06368)
cp2_2008_12_1...n1_tow_weekly 0.0009993  0.01578      0.968             (-0.02964, 0.03117)
cp2_2008_12_1...s1_tow_weekly   0.01771   0.0236      0.410             (-0.03527, 0.06176)
cp2_2008_12_1...n2_tow_weekly   0.01327  0.01728      0.456             (-0.02081, 0.04627)
cp2_2008_12_1...s2_tow_weekly   0.01849  0.02274      0.416             (-0.03267, 0.06078)
cp3_2009_01_1...n1_tow_weekly 0.0003183  0.01668      0.988             (-0.03223, 0.03124)
cp3_2009_01_1...s1_tow_weekly   0.02388   0.0233      0.294             (-0.02837, 0.06849)
cp3_2009_01_1...n2_tow_weekly   0.01069  0.01786      0.580             (-0.02557, 0.04208)
cp3_2009_01_1...s2_tow_weekly   0.02092  0.02247      0.342             (-0.02756, 0.06018)
cp4_2009_12_2...n1_tow_weekly   0.01304  0.03475      0.718               (-0.0519, 0.0836)
cp4_2009_12_2...s1_tow_weekly  0.009213  0.05383      0.832               (-0.1022, 0.1095)
cp4_2009_12_2...n2_tow_weekly  -0.03939   0.0404      0.320              (-0.1164, 0.04361)
cp4_2009_12_2...s2_tow_weekly   0.01732  0.04962      0.728              (-0.07626, 0.1096)
cp5_2010_01_2...n1_tow_weekly   0.01188  0.03793      0.774             (-0.05776, 0.09055)
cp5_2010_01_2...s1_tow_weekly   0.01714  0.05729      0.724              (-0.09892, 0.1267)
cp5_2010_01_2...n2_tow_weekly  -0.04501  0.04363      0.298              (-0.1307, 0.04531)
cp5_2010_01_2...s2_tow_weekly   0.02044  0.05283      0.694               (-0.0794, 0.1195)
cp6_2011_02_1...n1_tow_weekly  -0.01209   0.0478      0.820             (-0.09732, 0.08409)
cp6_2011_02_1...s1_tow_weekly   0.02295  0.06112      0.740              (-0.09219, 0.1329)
cp6_2011_02_1...n2_tow_weekly  -0.04252  0.05149      0.398              (-0.1408, 0.05717)
cp6_2011_02_1...s2_tow_weekly   0.01084  0.05892      0.844               (-0.1031, 0.1219)
cp7_2012_02_2...n1_tow_weekly  -0.03441  0.04709      0.470              (-0.1313, 0.05416)
cp7_2012_02_2...s1_tow_weekly  -0.02186  0.07134      0.744               (-0.1631, 0.1266)
cp7_2012_02_2...n2_tow_weekly  0.006063  0.05223      0.920               (-0.1016, 0.1055)
cp7_2012_02_2...s2_tow_weekly  -0.01174  0.06798      0.848               (-0.1468, 0.1126)
cp8_2012_04_0...n1_tow_weekly  -0.02055   0.0443      0.666              (-0.1169, 0.05902)
cp8_2012_04_0...s1_tow_weekly   -0.0134  0.06831      0.846               (-0.1442, 0.1262)
cp8_2012_04_0...n2_tow_weekly 0.0006358  0.05001      0.984             (-0.09702, 0.09352)
cp8_2012_04_0...s2_tow_weekly -0.009023  0.06481      0.876                (-0.142, 0.1068)
cp9_2013_03_0...n1_tow_weekly   0.07116  0.04662      0.122              (-0.03303, 0.1504)
cp9_2013_03_0...s1_tow_weekly  -0.01631  0.06346      0.788               (-0.148, 0.09769)
cp9_2013_03_0...n2_tow_weekly   0.01882  0.05313      0.702              (-0.08769, 0.1221)
cp9_2013_03_0...s2_tow_weekly -0.002872   0.0565      0.952               (-0.1139, 0.1133)
cp10_2014_01_...n1_tow_weekly   0.01698  0.05617      0.788              (-0.08713, 0.1269)
cp10_2014_01_...s1_tow_weekly -0.006531   0.1223      0.958               (-0.2431, 0.2445)
cp10_2014_01_...n2_tow_weekly   0.03562  0.07241      0.606               (-0.1054, 0.1746)
cp10_2014_01_...s2_tow_weekly  -0.02681    0.113      0.826                (-0.234, 0.1907)
              sin1_tow_weekly   0.04498  0.01919      0.020         *   (0.006032, 0.08248)
              cos1_tow_weekly    0.1489  0.02905     <2e-16       ***     (0.09309, 0.2085)
              sin2_tow_weekly  -0.02346  0.01887      0.212             (-0.06059, 0.01278)
              cos2_tow_weekly    0.1287   0.0288     <2e-16       ***     (0.07624, 0.1922)
              sin3_tow_weekly  -0.03121  0.01575      0.054         .  (-0.06139, 0.001313)
              cos3_tow_weekly   0.04595  0.02778      0.092         .    (-0.009125, 0.106)
             sin1_tom_monthly    0.1879  0.05808      0.002        **      (0.06514, 0.301)
             cos1_tom_monthly     0.378  0.05791     <2e-16       ***      (0.2562, 0.4747)
             sin2_tom_monthly    0.0567  0.02387      0.010         *   (0.006136, 0.09585)
             cos2_tom_monthly  -0.07404  0.02578      0.002        **   (-0.1265, -0.02131)
             sin3_tom_monthly  -0.03113  0.02474      0.226              (-0.08086, 0.0146)
             cos3_tom_monthly  -0.01881  0.02509      0.452              (-0.0671, 0.02805)
             sin4_tom_monthly -0.005921  0.02548      0.828                (-0.0517, 0.044)
             cos4_tom_monthly  -0.02279  0.02543      0.372              (-0.0735, 0.02907)
           sin1_toq_quarterly   0.08543  0.01964     <2e-16       ***     (0.04691, 0.1253)
           cos1_toq_quarterly -0.009899  0.02076      0.594             (-0.05187, 0.03149)
           sin2_toq_quarterly    0.1593  0.03523     <2e-16       ***     (0.08455, 0.2247)
           cos2_toq_quarterly   -0.1946  0.03192     <2e-16       ***    (-0.2509, -0.1307)
           sin3_toq_quarterly   0.04661  0.03784      0.210              (-0.02374, 0.1215)
           cos3_toq_quarterly  -0.06365  0.04223      0.140              (-0.1392, 0.02613)
           sin4_toq_quarterly  0.007778  0.02316      0.760              (-0.0361, 0.05307)
           cos4_toq_quarterly   -0.0794  0.02502      0.002        **    (-0.125, -0.02813)
              sin1_ct1_yearly   -0.2681  0.02571     <2e-16       ***    (-0.3194, -0.2202)
              cos1_ct1_yearly    0.6065  0.04778     <2e-16       ***      (0.5151, 0.6964)
              sin2_ct1_yearly   0.09868  0.02507     <2e-16       ***     (0.04927, 0.1443)
              cos2_ct1_yearly  -0.08284  0.02418     <2e-16       ***   (-0.1329, -0.03772)
              sin3_ct1_yearly    0.3063  0.02887     <2e-16       ***      (0.2511, 0.3658)
              cos3_ct1_yearly   0.09213  0.02617     <2e-16       ***     (0.04353, 0.1421)
              sin4_ct1_yearly   0.08335  0.02342     <2e-16       ***     (0.03243, 0.1274)
              cos4_ct1_yearly  -0.05211  0.01869      0.004        **   (-0.0907, -0.01711)
              sin5_ct1_yearly   -0.1368  0.02605     <2e-16       ***    (-0.1927, -0.0885)
              cos5_ct1_yearly    -0.106  0.02239     <2e-16       ***   (-0.1481, -0.06079)
              sin6_ct1_yearly   -0.1869  0.02364     <2e-16       ***    (-0.2319, -0.1435)
              cos6_ct1_yearly   -0.1542  0.02602     <2e-16       ***    (-0.2025, -0.1051)
              sin7_ct1_yearly   -0.1481  0.02447     <2e-16       ***   (-0.1926, -0.09977)
              cos7_ct1_yearly   0.03018  0.02485      0.232             (-0.01814, 0.07927)
              sin8_ct1_yearly    0.1709  0.03515     <2e-16       ***     (0.09638, 0.2321)
              cos8_ct1_yearly   0.09833  0.03261      0.002        **     (0.04019, 0.1671)
              sin9_ct1_yearly   0.02573  0.02528      0.312             (-0.01985, 0.07604)
              cos9_ct1_yearly  -0.04296  0.02424      0.074         .  (-0.08891, 0.002212)
             sin10_ct1_yearly   -0.1238  0.02437     <2e-16       ***   (-0.1683, -0.07958)
             cos10_ct1_yearly   -0.1818  0.02648     <2e-16       ***     (-0.238, -0.1378)
             sin11_ct1_yearly  -0.06817  0.02659      0.012         *   (-0.1206, -0.01333)
             cos11_ct1_yearly  -0.05662  0.02529      0.028         *  (-0.1065, -0.007213)
             sin12_ct1_yearly     0.323  0.06423     <2e-16       ***      (0.1925, 0.4424)
             cos12_ct1_yearly  -0.07043  0.05812      0.228                (-0.1805, 0.044)
             sin13_ct1_yearly  -0.04137  0.02426      0.090         .  (-0.08939, 0.005005)
             cos13_ct1_yearly    0.1126  0.02524     <2e-16       ***     (0.06284, 0.1634)
             sin14_ct1_yearly    0.0695  0.02321     <2e-16       ***     (0.02682, 0.1138)
             cos14_ct1_yearly  0.007866  0.02524      0.754             (-0.04384, 0.05707)
             sin15_ct1_yearly   0.06504  0.02431      0.004        **     (0.01984, 0.1146)
             cos15_ct1_yearly  -0.05512  0.02555      0.042         *  (-0.103, -0.0007661)
                       y_lag7    0.7971   0.1588     <2e-16       ***       (0.4651, 1.049)
                       y_lag8    0.4709   0.1457     <2e-16       ***      (0.2216, 0.7792)
                       y_lag9  -0.00889   0.1254      0.946               (-0.2659, 0.2306)
             y_avglag_7_14_21     1.504   0.1605     <2e-16       ***        (1.209, 1.838)
             y_avglag_7_to_13     0.433   0.1393      0.004        **        (0.155, 0.707)
            y_avglag_14_to_20    0.1228   0.1017      0.208              (-0.07056, 0.3544)
Signif. Code: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Multiple R-squared: 0.7276,   Adjusted R-squared: 0.717
F-statistic: 66.69 on 110 and 2830 DF,   p-value: 1.110e-16
Model AIC: 18935.0,   model BIC: 19600.0

WARNING: the condition number is large, 2.56e+04. This might indicate that there are strong multicollinearity or other numerical problems.
WARNING: the F-ratio and its p-value on regularized methods might be misleading, they are provided only for reference purposes.
215 # Check component plot of SK_2 on second fold
216 model = bm.result["SK_2"]["rolling_evaluation"]["split_1"]["pipeline_result"].model
217 fig = model[-1].plot_components()
218 plotly.io.show(fig)

Compare forecasts

To obtain forecasts run the extract_forecasts method. You only need to run this once.

226 bm.extract_forecasts()

This method does two things.

  • For every config, it gathers forecast results across rolling windows and stores it as a dataframe in rolling_forecast_df under the config key. This helps in comparing forecasts and prediction accuracy across splits for the config.

235 # Forecast across rolling windows for SK_1
236 forecast_sk_1 = bm.result["SK_1"]["rolling_forecast_df"]
237 forecast_sk_1.head()
train_end_date forecast_step split_num ts actual forecast forecast_lower forecast_upper
0 2015-12-29 1 0 2015-12-30 8.391403 8.082386 7.315174 8.849599
1 2015-12-29 2 0 2015-12-31 8.004700 7.948438 7.209923 8.686953
2 2015-12-29 3 0 2016-01-01 7.589336 7.864888 6.989538 8.740238
3 2015-12-29 4 0 2016-01-02 7.825245 7.664048 6.877162 8.450933
4 2015-12-29 5 0 2016-01-03 8.249314 8.570844 7.569269 9.572420


  • Concatenates rolling_forecast_df for all the configs and stores it as a dataframe in the class attribute forecasts. This helps in comparing forecasts and prediction accuracies across configs.

243 # Forecasts across configs
244 bm.forecasts.head()
train_end_date forecast_step split_num ts actual SK_1_forecast SK_1_forecast_lower SK_1_forecast_upper SK_2_forecast SK_2_forecast_lower SK_2_forecast_upper
0 2015-12-29 1 0 2015-12-30 8.391403 8.082386 7.315174 8.849599 8.123937 7.370882 8.876992
1 2015-12-29 2 0 2015-12-31 8.004700 7.948438 7.209923 8.686953 7.981728 7.248408 8.715049
2 2015-12-29 3 0 2016-01-01 7.589336 7.864888 6.989538 8.740238 7.830524 6.967410 8.693638
3 2015-12-29 4 0 2016-01-02 7.825245 7.664048 6.877162 8.450933 7.718439 6.938497 8.498380
4 2015-12-29 5 0 2016-01-03 8.249314 8.570844 7.569269 9.572420 8.483205 7.493646 9.472763


For any config you can plot forecasts across splits. This allows you to quickly check if there is any particular time window where the test performance drops. The forecasts for adjacent folds will overlap if the time windows of the corresponding folds overlap.

251 fig = bm.plot_forecasts_by_config(config_name="SK_1")
252 plotly.io.show(fig)

The importance of this function becomes more significant when assessing a models performance over a longer period e.g. a year or multiple years. You can quickly catch if models test performance drops during weekends, specific months or holiday seasons.

You can also compare forecasts from multiple configs by forecast_step which is defined as any number between 1 and forecast_horizon. This is useful in forecasts with longer forecast horizons to check if the forecast volatility changes over time.

263 fig = bm.plot_forecasts_by_step(forecast_step=3)
264 plotly.io.show(fig)

Compare Errors

You can compare the predictive performance of your models via multiple evaluation metrics. In this example we will use MAPE and RMSE, but you can use any metric from EvaluationMetricEnum.

272 metric_dict = {
273     "MAPE": EvaluationMetricEnum.MeanAbsolutePercentError,
274     "RMSE": EvaluationMetricEnum.RootMeanSquaredError
275 }

Non Grouping Errors

To compare evaluation metrics without any grouping use get_evaluation_metrics. The output shows metric values by config and split. We can group by config_name to get metric values aggregated across all folds.

284 # Compute evaluation metrics
285 evaluation_metrics_df = bm.get_evaluation_metrics(metric_dict=metric_dict)
286 # Aggregate by model across splits
287 error_df = evaluation_metrics_df.drop(columns=["split_num"]).groupby("config_name").mean()
288 error_df
train_MAPE test_MAPE train_RMSE test_RMSE
config_name
SK_1 3.578636 4.565086 0.444640 0.439975
SK_2 3.559621 4.337858 0.441185 0.427289


292 # Visualize
293 fig = bm.plot_evaluation_metrics(metric_dict)
294 plotly.io.show(fig)

Train MAPE is high because some values in training dataset are close to 0.

You can also compare the predictive accuracy across splits for any model from configs. This allows you to check if the model performance varies significantly across time periods.

302 # Compute evaluation metrics for a single config
303 evaluation_metrics_df = bm.get_evaluation_metrics(metric_dict=metric_dict, config_names=["SK_1"])
304 # Aggregate by split number
305 error_df = evaluation_metrics_df.groupby("split_num").mean()
306 error_df.head()
train_MAPE test_MAPE train_RMSE test_RMSE
split_num
0 3.579761 2.984356 0.444592 0.270273
1 3.559130 2.080209 0.442756 0.213116
2 3.588503 6.247890 0.445517 0.555328
3 3.587149 6.947891 0.445697 0.721184


310 # Visualize
311 title = "Average evaluation metric across rolling windows"
312 data = []
313 # Each row (index) is a config. Adds each row to the bar plot.
314 for index in error_df.index:
315     data.append(
316         go.Bar(
317             name=index,
318             x=error_df.columns,
319             y=error_df.loc[index].values
320         )
321     )
322 layout = go.Layout(
323     xaxis=dict(title=None),
324     yaxis=dict(title="Metric Value"),
325     title=title,
326     title_x=0.5,
327     showlegend=True,
328     barmode="group",
329 )
330 fig = go.Figure(data=data, layout=layout)
331 plotly.io.show(fig)

Grouping Errors

To compare evaluation metrics with grouping use get_grouping_evaluation_metrics. This allows you to group the error values by time features such as day of week, month etc.

339 # Compute grouped evaluation metrics
340 grouped_evaluation_df = bm.get_grouping_evaluation_metrics(
341     metric_dict=metric_dict,
342     which="test",
343     groupby_time_feature="str_dow")
344 # Aggregate by split number
345 error_df = grouped_evaluation_df.groupby(["str_dow", "config_name"]).mean()
346 error_df
split_num test MAPE test RMSE
str_dow config_name
1-Mon SK_1 1.5 7.105198 0.676760
SK_2 1.5 7.460640 0.710098
2-Tue SK_1 1.5 5.837785 0.496679
SK_2 1.5 5.664220 0.481733
3-Wed SK_1 1.5 4.645936 0.392558
SK_2 1.5 4.430198 0.374260
4-Thu SK_1 1.5 3.371199 0.270771
SK_2 1.5 2.940572 0.236874
5-Fri SK_1 1.5 3.925368 0.311203
SK_2 1.5 3.101233 0.245622
6-Sat SK_1 1.5 2.368099 0.184806
SK_2 1.5 2.039809 0.159080
7-Sun SK_1 1.5 4.702020 0.410814
SK_2 1.5 4.728331 0.414808


350 # Visualize
351 fig = bm.plot_grouping_evaluation_metrics(
352     metric_dict=metric_dict,
353     which="test",
354     groupby_time_feature="str_dow")
355 plotly.io.show(fig)

As you can see all the models have higher MAPE and RMSE during weekends. That means adding is_weekend indicator to the models will help.

Compare runtimes

You can compare and visualize runtimes of the models using the following codes.

365 # Compute runtimes
366 runtime_df = bm.get_runtimes()
367 # Aggregate across splits
368 runtimes_df = runtime_df.drop(columns=["split_num"]).groupby("config_name").mean()
369 runtimes_df
runtime_sec
config_name
SK_1 35.45075
SK_2 29.08700


373 # Visualize
374 fig = bm.plot_runtimes()
375 plotly.io.show(fig)

You can see Silverkite models run almost 3 times faster compared to Prophet.

Debugging the Benchmark

When the run method is called, the input configs are first assessed of their suitability for a cohesive benchmarking procedure via the validate method. This is done prior to passing the configs to the forecasting pipeline to save wasted computing time for the user. Though not necessary, the user is encouraged to use validate for debugging.

The validate method runs a series of checks to ensure that

  • The configs are compatible among themselves. For example, it checks if all the configs have the same forecast horizon.

  • The configs are compatible with the CV schema. For example, forecast_horizon and periods_between_train_test parameters of configs are matched against that of the tscv.

Note that the validate method does not guarantee that the models will execute properly while in the pipeline. It is a good idea to do a test run on a smaller data and/ or smaller number of splits before running the full procedure.

In the event of a mismatch a ValueError is raised with informative error messages to help the user in debugging. Some examples are provided below.

Error due to incompatible model components in config

406 # some_regressor is not part of Prophet's model components
407 model_components=ModelComponentsParam(
408     regressors={
409         "some_regressor": ["regressor1", "regressor2", "regressor_categ"]
410     }
411 )
412 invalid_prophet = replace(Silverkite_1, model_components_param=model_components)
413 invalid_configs = {"invalid_prophet": invalid_prophet}
414 bm = BenchmarkForecastConfig(df=df, configs=invalid_configs, tscv=tscv)
415 try:
416     bm.validate()
417 except ValueError as err:
418     print(err)

Out:

Unexpected key(s) found: {'some_regressor'}. The valid keys are: dict_keys(['regressor_cols'])

Error due to wrong template name

424 # model template name is not part of TemplateEnum, thus invalid
425 unknown_template = replace(Silverkite_1, model_template="SOME_TEMPLATE")
426 invalid_configs = {"unknown_template": unknown_template}
427 bm = BenchmarkForecastConfig(df=df, configs=invalid_configs, tscv=tscv)
428 try:
429     bm.validate()
430 except ValueError as err:
431     print(err)

Out:

Model Template 'SOME_TEMPLATE' is not recognized! Must be one of: SILVERKITE, SILVERKITE_DAILY_1_CONFIG_1, SILVERKITE_DAILY_1_CONFIG_2, SILVERKITE_DAILY_1_CONFIG_3, SILVERKITE_DAILY_1, SILVERKITE_DAILY_90, SILVERKITE_WEEKLY, SILVERKITE_MONTHLY, SILVERKITE_HOURLY_1, SILVERKITE_HOURLY_24, SILVERKITE_HOURLY_168, SILVERKITE_HOURLY_336, SILVERKITE_EMPTY, SK, PROPHET, AUTO_ARIMA, SILVERKITE_TWO_STAGE, MULTISTAGE_EMPTY, AUTO, LAG_BASED, SILVERKITE_WOW or satisfy the `SimpleSilverkiteTemplate` rules.

Error due to different forecast horizons in configs

437 # the configs are valid by themselves, however incompatible for
438 # benchmarking as these have different forecast horizons
439 Silverkite_forecast_horizon_30 = replace(Silverkite_1, forecast_horizon=30)
440 invalid_configs = {
441     "Silverkite": Silverkite_1,
442     "Silverkite_30": Silverkite_forecast_horizon_30
443 }
444 bm = BenchmarkForecastConfig(df=df, configs=invalid_configs, tscv=tscv)
445 try:
446     bm.validate()
447 except ValueError as err:
448     print(err)

Out:

Silverkite_30's 'forecast_horizon' (30) does not match that of 'tscv' (7).

Error due to different forecast horizons in config and tscv

454 # Error due to different forecast horizons in config and tscv
455 tscv = RollingTimeSeriesSplit(forecast_horizon=15)
456 bm = BenchmarkForecastConfig(df=df, configs=configs, tscv=tscv)
457 try:
458     bm.validate()
459 except ValueError as err:
460     print(err)

Out:

SK_1's 'forecast_horizon' (7) does not match that of 'tscv' (15).

Total running time of the script: ( 4 minutes 43.714 seconds)

Gallery generated by Sphinx-Gallery