Forecast One By One

A useful feature for short-term forecast in Silverkite model family is autoregression. Silverkite has an “auto” option for autoregression, which automatically selects the autoregression lag orders based on the data frequency and forecast horizons. One important rule of this “auto” option is that the minimum order of autoregression terms is at least the forecast horizon. For example, if the forecast horizon is 3 on a daily model, the minimum order of autoregression is set to 3. The “auto” option won’t have an order of 2 in this case, because the 3rd day forecast will need the 1st day’s observation, which isn’t available at the current time. Although the model can make predictions with an autoregression lag order less than the forecast horizon via simulations, it takes longer time to run and is not the preferred behavior in the “auto” option.

However, in many cases, using smaller autoregression lag orders can give more accurate forecast results. We observe that the only barrier of using an autoregression term of order 2 in the 3-day forecast model is the 3rd day, while we can use it freely for the first 2 days. Similarly, we are able to use an autoregression term of order 1 for the 1st day. In a 3 day forecast, if the accuracy of all 3 days are important, then replacing the first 2 days’ models with shorter autoregression lag orders can improve the accuracy. The forecast-one-by-one algorithm is designed in this context.

The observations above together bring the idea of the forecast-one-by-one algorithm. The algorithm allows fitting multiple models with the “auto” option in autoregression, when one is forecasting with a forecast horizon longer than 1. For each model, the “auto” option for autoregression selects the smallest available autoregression lag order and predicts for the corresponding forecast steps, thus improving the forecast accuracy for the early steps.

In this example, we will cover how to activate the forecast-one-by-one approach via the ForecastConfig and the Forecaster classes. For a detailed API reference, please see the ForecastConfig and OneByOneEstimator classes.

40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 import warnings

 warnings.filterwarnings("ignore")

 import plotly
 from greykite.common.data_loader import DataLoader
 from greykite.framework.templates.autogen.forecast_config import ForecastConfig
 from greykite.framework.templates.autogen.forecast_config import ModelComponentsParam
 from greykite.framework.templates.forecaster import Forecaster
 from greykite.framework.templates.model_templates import ModelTemplateEnum
 from greykite.framework.utils.result_summary import summarize_grid_search_results

 # Loads dataset into pandas DataFrame
 dl = DataLoader()
 df = dl.load_peyton_manning()

The forecast-one-by-one option

The forecast-one-by-one option is specified through the forecast_one_by_one parameter in ForecastConfig.

63
64
65
66
67
68
69
70
 config = ForecastConfig(
     model_template=ModelTemplateEnum.SILVERKITE.name,
     forecast_horizon=3,
     model_components_param=ModelComponentsParam(
         autoregression=dict(autoreg_dict="auto")
     ),
     forecast_one_by_one=True
 )

The forecast_one_by_one parameter can be specified in the following ways

  • ``True``: every forecast step will be a separate model. The number of models equals the forecast horizon. In this example, 3 models will be fit with the 3 forecast steps.

  • ``False``: the forecast-one-by-one method is turned off. This is the default behavior and a single model is used for all forecast steps.

  • A list of integers: each integer corresponds to a model, and it is the number of steps. For example, in a 7 day forecast, specifying forecast_one_by_one=[1, 2, 4] will result in 3 models. The first model forecasts the 1st day with forecast horizon 1; The second model forecasts the 2nd - 3rd days with forecast horizon 3; The third model forecasts the 4th - 7th days with forecast horizon 7. In this case, the sum of the list entries must equal the forecast horizon.

  • an integer ``n``: every model will account for n steps. The last model will account for the rest <n steps. For example in a 7 day forecast, specifying forecast_one_by_one=2 will result in 4 models, which is equivalent to forecast_one_by_one=[2, 2, 2, 1].

Note

forecast_one_by_one is activated only when there are parameters in the model that depend on the forecast horizon. Currently the only parameter that depends on forecast horizon is autoreg_dict="auto". If you do not specify autoreg_dict="auto", the forecast_one_by_one parameter will be ignored.

Note

Forecast-one-by-one fits multiple models to increase accuracy, which may cause the training time to increase linearly with the number of models. Please make sure your forecast_one_by_one parameter and forecast horizon result in a reasonable number of models.

Next, let’s run the model and look at the result.

106
107
108
109
110
111
 # Runs the forecast
 forecaster = Forecaster()
 result = forecaster.run_forecast_config(
     df=df.iloc[-365:].reset_index(drop=True),  # Uses less data to speed up this example.
     config=config
 )

Out:

Fitting 3 folds for each of 1 candidates, totalling 3 fits

You may see a few warnings like “The future x length is 0, which doesn’t match the model forecast horizon 3, using only the model with the longest forecast horizon for prediction.” This is an expected behavior when calculating the training errors. Because the models are mapped to the forecast period only, but not to the training period. Therefore, only the last model is used to get the fitted values on the training period. You don’t need to worry about it.

Everything on the forecast_result level is the same as not activating forecast-one-by-one. For example, we can view the cross-validation results in the same way.

126
127
128
129
130
131
132
133
134
135
 # Summarizes the CV results
 cv_results = summarize_grid_search_results(
     grid_search=result.grid_search,
     decimals=1,
     # The below saves space in the printed output. Remove to show all available metrics and columns.
     cv_report_metrics=None,
     column_order=["rank", "mean_test", "split_test", "mean_train", "split_train", "mean_fit_time", "mean_score_time", "params"])
 cv_results["params"] = cv_results["params"].astype(str)
 cv_results.set_index("params", drop=True, inplace=True)
 cv_results.transpose()
params []
rank_test_MAPE 1
mean_test_MAPE 5.7
split_test_MAPE (11.2, 1.8, 3.9)
mean_train_MAPE 2.9
split_train_MAPE (1.8, 3.5, 3.5)
mean_fit_time 16.9
mean_score_time 2.9


When you need to access estimator level attributes, for example, model summary or component plots, the returned result will be a list of the original type, because we fit multiple models. The model summary list can be accessed in the same way and you can use index to get the model summary for a single model.

143
144
145
146
147
 # Gets the model summary list
 one_by_one_estimator = result.model[-1]
 summaries = one_by_one_estimator.summary()
 # Prints the model summary for 1st model only
 print(summaries[0])

Out:

================================ Model Summary =================================

Number of observations: 367,   Number of features: 98
Method: Ridge regression
Number of nonzero features: 98
Regularization parameter: 4.535

Residuals:
         Min           1Q       Median           3Q          Max
     -0.8717      -0.1678     -0.05206      0.09778        3.193

             Pred_col    Estimate Std. Err Pr(>)_boot sig. code                   95%CI
            Intercept        1.03   0.2604     <2e-16       ***          (0.4786, 1.47)
  events_C...New Year     0.08146  0.06858      0.218                (-0.04339, 0.2034)
  events_C...w Year-1    -0.03677  0.03686      0.298                (-0.1134, 0.03041)
  events_C...w Year-2    -0.07126  0.04769      0.120                (-0.1599, 0.01076)
  events_C...w Year+1     -0.0126   0.0783      0.832                 (-0.1506, 0.1384)
  events_C...w Year+2    -0.03909  0.08776      0.708                  (-0.1877, 0.125)
 events_Christmas Day     -0.0698  0.05615      0.120                      (-0.179, 0.)
  events_C...as Day-1     -0.1154  0.08295      0.076         .           (-0.2513, 0.)
  events_C...as Day-2    -0.02082  0.02119      0.218              (-0.06451, 0.005532)
  events_C...as Day+1    -0.03133  0.02996      0.170             (-0.09972, 1.975e-05)
  events_C...as Day+2      0.1778   0.1267      0.090         .            (0., 0.3887)
  events_E...Ireland]    0.002869  0.01986      0.600               (-0.04315, 0.04583)
  events_E...eland]-1    -0.02038  0.01931      0.186               (-0.0597, 0.003384)
  events_E...eland]-2     0.00615  0.01398      0.470               (-0.02183, 0.03585)
  events_E...eland]+1    -0.01509   0.0178      0.266               (-0.05209, 0.01381)
  events_E...eland]+2    0.009114  0.01462      0.344               (-0.01345, 0.04457)
   events_Good Friday     0.02176    0.025      0.250               (-0.01066, 0.07891)
 events_Good Friday-1    -0.03552  0.03225      0.120                     (-0.1172, 0.)
 events_Good Friday-2    -0.01343  0.01889      0.348               (-0.05603, 0.02372)
 events_Good Friday+1     0.00615  0.01398      0.470               (-0.02183, 0.03585)
 events_Good Friday+2    -0.02038  0.01931      0.186               (-0.0597, 0.003384)
  events_I...ence Day     0.06223  0.04274      0.092         .     (-0.003515, 0.1379)
  events_I...ce Day-1    -0.07515  0.04863      0.082         .    (-0.1624, 0.0001842)
  events_I...ce Day-2   0.0004795  0.02601      0.992                (-0.05914, 0.0483)
  events_I...ce Day+1    -0.07822  0.04252      0.048         *           (-0.1586, 0.)
  events_I...ce Day+2    -0.04327    0.032      0.118               (-0.1114, 0.004112)
     events_Labor Day    -0.01627  0.04983      0.714                (-0.1221, 0.06818)
   events_Labor Day-1  -0.0008638   0.0435      0.840               (-0.08271, 0.07742)
   events_Labor Day-2     0.03416  0.02687      0.142                 (-0.014, 0.08467)
   events_Labor Day+1     0.01442   0.0303      0.564               (-0.04824, 0.07184)
   events_Labor Day+2     0.01527  0.06051      0.896                (-0.09276, 0.1341)
  events_Memorial Day     -0.1105  0.08152      0.094         .           (-0.2541, 0.)
  events_M...al Day-1    -0.04992  0.04245      0.114                     (-0.1435, 0.)
  events_M...al Day-2      0.0142  0.02363      0.322                (-0.03067, 0.0703)
  events_M...al Day+1   -0.004748  0.01838      0.874                (-0.0458, 0.02929)
  events_M...al Day+2     0.01526  0.02001      0.284               (-0.01359, 0.05859)
 events_New Years Day     -0.0556  0.04115      0.092         .           (-0.1323, 0.)
  events_N...rs Day-1   -0.009647  0.02152      0.820               (-0.06441, 0.02407)
  events_N...rs Day-2    -0.03415  0.03001      0.150              (-0.09489, 0.001054)
  events_N...rs Day+1     0.08857  0.06066      0.074         .            (0., 0.1919)
  events_N...rs Day+2   -0.007288  0.02469      0.466               (-0.06169, 0.04518)
         events_Other      0.1168  0.07999      0.148                (-0.02647, 0.2905)
       events_Other-1    -0.06767   0.0479      0.130                (-0.1568, 0.03284)
       events_Other-2     0.02303  0.04163      0.576                (-0.04767, 0.1061)
       events_Other+1    0.006867  0.04744      0.910                 (-0.08353, 0.104)
       events_Other+2    -0.02896  0.04465      0.526                (-0.1167, 0.05925)
  events_Thanksgiving    -0.04715  0.04028      0.116                     (-0.1362, 0.)
  events_T...giving-1   -0.009419  0.01666      0.320               (-0.05329, 0.01504)
  events_T...giving-2    -0.02868  0.02891      0.180                    (-0.09503, 0.)
  events_T...giving+1     0.03657  0.03073      0.164               (-0.004248, 0.0948)
  events_T...giving+2    -0.02721  0.02693      0.164             (-0.08967, 0.0009159)
  events_Veterans Day    -0.02841  0.03068      0.188                      (-0.107, 0.)
  events_V...ns Day-1     -0.0652  0.04967      0.100                     (-0.1606, 0.)
  events_V...ns Day-2     0.02566  0.03446      0.294                (-0.03139, 0.1045)
  events_V...ns Day+1    0.009204  0.01774      0.416               (-0.02502, 0.04603)
  events_V...ns Day+2    -0.05458  0.04205      0.094         .           (-0.1393, 0.)
        str_dow_2-Tue     0.01025  0.01192      0.394               (-0.01139, 0.03335)
        str_dow_3-Wed      0.0198  0.01105      0.070         .     (-0.00106, 0.04092)
        str_dow_4-Thu     0.02201  0.01438      0.128              (-0.006094, 0.04906)
        str_dow_5-Fri  -0.0003953  0.01025      0.980               (-0.02048, 0.01872)
        str_dow_6-Sat    -0.05369  0.01254     <2e-16       ***    (-0.07822, -0.02964)
        str_dow_7-Sun    -0.02082  0.01608      0.180                (-0.05093, 0.0136)
                  ct1      0.2615  0.07261      0.002        **        (0.1335, 0.4153)
       is_weekend:ct1     0.08703  0.03217      0.008        **        (0.02198, 0.144)
    str_dow_2-Tue:ct1    -0.04351  0.05595      0.428                (-0.1618, 0.05925)
    str_dow_3-Wed:ct1     0.01914  0.03885      0.630               (-0.04798, 0.09837)
    str_dow_4-Thu:ct1     -0.0462  0.04072      0.246                (-0.1238, 0.03585)
    str_dow_5-Fri:ct1      0.0709  0.03755      0.064         .     (-0.005861, 0.1419)
    str_dow_6-Sat:ct1    -0.05695  0.04296      0.196                (-0.1225, 0.03774)
    str_dow_7-Sun:ct1       0.144  0.06054      0.018         *    (-0.0003027, 0.2438)
  ct1:sin1_tow_weekly     -0.1232  0.04029      0.006        **     (-0.1999, -0.03916)
  ct1:cos1_tow_weekly       0.223  0.08543      0.006        **        (0.0652, 0.3887)
  ct1:sin2_tow_weekly     -0.1243  0.04916      0.014         *     (-0.2156, -0.02182)
  ct1:cos2_tow_weekly      0.2013  0.07443      0.008        **       (0.06944, 0.3585)
      sin1_tow_weekly      0.1057  0.02879     <2e-16       ***       (0.04846, 0.1641)
      cos1_tow_weekly    0.004318  0.03939      0.930               (-0.06834, 0.08089)
      sin2_tow_weekly    -0.01912  0.02945      0.514               (-0.07778, 0.03428)
      cos2_tow_weekly     0.06921  0.03491      0.040         *       (0.002263, 0.136)
      sin3_tow_weekly    -0.02214  0.01086      0.056         .  (-0.04558, -0.0007143)
      cos3_tow_weekly    0.006428  0.01342      0.650               (-0.02012, 0.03262)
      sin4_tow_weekly     0.02214  0.01086      0.056         .    (0.0007143, 0.04558)
      cos4_tow_weekly    0.006428  0.01342      0.650               (-0.02012, 0.03262)
   sin1_toq_quarterly     0.01107   0.0263      0.708               (-0.04502, 0.05348)
   cos1_toq_quarterly    -0.02902  0.02945      0.282               (-0.08998, 0.03087)
   sin2_toq_quarterly    0.001636  0.02793      0.940               (-0.04719, 0.06075)
   cos2_toq_quarterly   -0.003307  0.02514      0.892               (-0.05551, 0.04298)
   sin3_toq_quarterly    -0.03133  0.02302      0.180               (-0.07384, 0.01158)
   cos3_toq_quarterly     0.00307   0.0309      0.938                (-0.05386, 0.0652)
   sin4_toq_quarterly    -0.02584  0.02448      0.302               (-0.07321, 0.02293)
   cos4_toq_quarterly    -0.02585  0.02509      0.308               (-0.07578, 0.02105)
   sin5_toq_quarterly    -0.03795  0.03189      0.210               (-0.08919, 0.03401)
   cos5_toq_quarterly     0.01279  0.02433      0.582               (-0.03775, 0.06203)
               y_lag1      0.5956  0.07784     <2e-16       ***        (0.4133, 0.7081)
               y_lag2      -0.115  0.07453      0.124                (-0.2206, 0.05285)
               y_lag3     0.02878  0.05422      0.590                (-0.08187, 0.1284)
     y_avglag_7_14_21      0.2251  0.06835     <2e-16       ***        (0.1048, 0.3705)
      y_avglag_1_to_7     0.06802   0.0871      0.410                  (-0.104, 0.2386)
     y_avglag_8_to_14     0.04266  0.05207      0.380                (-0.06847, 0.1441)
Signif. Code: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Multiple R-squared: 0.7745,   Adjusted R-squared: 0.7475
F-statistic: 27.179 on 39 and 326 DF,   p-value: 1.110e-16
Model AIC: 1462.8,   model BIC: 1619.8

WARNING: the condition number is large, 2.94e+04. This might indicate that there are strong multicollinearity or other numerical problems.
WARNING: the F-ratio and its p-value on regularized methods might be misleading, they are provided only for reference purposes.

We can access the component plots in a similar way.

152
153
154
155
 # Gets the fig list
 figs = one_by_one_estimator.plot_components()
 # Shows the component plot for 1st model only
 plotly.io.show(figs[0])

Total running time of the script: ( 1 minutes 43.393 seconds)

Gallery generated by Sphinx-Gallery