Tune your first forecast model

This is a basic tutorial for creating and tuning a forecast model. It is intended to provide a basic sense of a forecast process without assuming background knowledge in forecasting.

You can use the PROPHET or SILVERKITE model. In this tutorial, we focus on SILVERKITE. However, the basic ideas of tuning are similar to both models. You may see detailed information about PROPHET at Prophet.

SILVERKITE decomposes time series into various components, and it creates time-based features, autoregressive features, together with user-provided features such as macro-economic features and their interactions, then performs a machine learning regression model to learn the relationship between the time series and these features. The forecast is based on the learned relationship and the future values of these features. Therefore, including the correct features is the key to success.

Common features include:

Datetime derivatives:

Including features derived from datetime such as day of year, hour of day, weekday, is_weekend and etc. These features are useful in capturing special patterns. For example, the patterns of weekdays and weekends are different for most business related time series, and this can be modeled with is_weekend.

Growth:

First defines the basic feature ct1 that counts how long has passed in terms of years (could be fraction) since the first day of training data. For example, if the training data starts with “2018-01-01”, then the date has ct1=0.0, and “2018-01-02” has ct1=1/365. “2019-01-01” has ct1=1.0. This ct1 can be as granular as needed. A separate growth function can be applied to ct1 to support different types of growth model. For example, ct2 is defined as the square of ct1 to model quadratic growth.

Trend:

Trend describes the average tendency of the time series. It is defined through the growth term with possible changepoints. At every changepoint, the growth rate could change (faster or slower). For example, if ct1 (linear growth) is used with changepoints, the trend is modeled as piece-wise linear.

Seasonality:

Seasonality describes the periodical pattern of the time series. It contains multiple levels including daily seasonality, weekly seasonality, monthly seasonality, quarterly seasonality and yearly seasonality. Seasonality are defined through Fourier series with different orders. The greater the order, the more detailed periodical pattern the model can learn. However, an order that is too large can lead to overfitting.

Events:

Events include holidays and other short-term occurrences that could temporarily affect the time series, such as Thanksgiving long weekend. Typically, events are regular and repeat at know times in the future. These features made of indicators that covers the event day and their neighbor days.

Autoregression:

Autoregressive features include the time series observations in the past and their aggregations. For example, the past day’s observation, the same weekday on the past week, or the average of the past 7 days, etc. can be used. Note that autoregression features are very useful in short term forecasts, however, this should be avoided in long term forecast. The reason is that long-term forecast focuses more on the correctness of trend, seasonality and events. The lags and autoregressive terms in a long-term forecast are calculated based on the forecasted values. The further we forecast into the future, the more forecasted values we need to create the autoregressive terms, making the forecast less stable.

Custom:

Extra features that are relevant to the time series such as macro-ecomonic features that are expected to affect the time series. Note that these features need to be manually provided for both the training and forecasting periods.

Interactions:

Any interaction between the features above.

Now let’s use an example to go through the full forecasting and tuning process. In this example, we’ll load a dataset representing log(daily page views) on the Wikipedia page for Peyton Manning. It contains values from 2007-12-10 to 2016-01-20. More dataset info here.

 import datetime

 import numpy as np
 import pandas as pd
 import plotly

 from greykite.algo.changepoint.adalasso.changepoint_detector import ChangepointDetector
 from greykite.algo.forecast.silverkite.constants.silverkite_holiday import SilverkiteHoliday
 from greykite.algo.forecast.silverkite.constants.silverkite_seasonality import SilverkiteSeasonalityEnum
 from greykite.algo.forecast.silverkite.forecast_simple_silverkite_helper import cols_interact
 from greykite.common import constants as cst
 from greykite.common.features.timeseries_features import build_time_features_df
 from greykite.common.features.timeseries_features import convert_date_to_continuous_time
 from greykite.framework.benchmark.data_loader_ts import DataLoaderTS
 from greykite.framework.templates.autogen.forecast_config import EvaluationPeriodParam
 from greykite.framework.templates.autogen.forecast_config import ForecastConfig
 from greykite.framework.templates.autogen.forecast_config import MetadataParam
 from greykite.framework.templates.autogen.forecast_config import ModelComponentsParam
 from greykite.framework.templates.forecaster import Forecaster
 from greykite.framework.templates.model_templates import ModelTemplateEnum
 from greykite.framework.utils.result_summary import summarize_grid_search_results


 # Loads dataset into UnivariateTimeSeries
 dl = DataLoaderTS()
 ts = dl.load_peyton_manning_ts()
 df = ts.df  # cleaned pandas.DataFrame

Exploratory data analysis (EDA)

After reading in a time series, we could first do some exploratory data analysis. The UnivariateTimeSeries class is used to store a timeseries and perform EDA.

 # describe
 print(ts.describe_time_col())
 print(ts.describe_value_col())

Out:

{'data_points': 2964, 'mean_increment_secs': 86400.0, 'min_timestamp': Timestamp('2007-12-10 00:00:00'), 'max_timestamp': Timestamp('2016-01-20 00:00:00')}
count    2905.000000
mean        8.138958
std         0.845957
min         5.262690
25%         7.514800
50%         7.997999
75%         8.580168
max        12.846747
Name: y, dtype: float64

The df has two columns, time column “ts” and value column “y”. The data is daily that ranges from 2007-12-10 to 2016-01-20. The data value ranges from 5.26 to 12.84

Let’s plot the original timeseries. (The interactive plot is generated by plotly: click to zoom!)

 fig = ts.plot()
 plotly.io.show(fig)

A few exploratory plots can be plotted to reveal the time series’s properties. The UnivariateTimeSeries class has a very powerful plotting tool plot_quantiles_and_overlays. A tutorial of using the function can be found at Seasonality Plots.

Baseline model

A simple forecast can be created on the data set, see details in Simple Forecast. Note that if you do not provide any extra parameters, all model parameters are by default. The default parameters are chosen conservatively, so consider this a baseline model to assess forecast difficulty and make further improvements if necessary.

 # Specifies dataset information
 metadata = MetadataParam(
     time_col="ts",  # name of the time column
     value_col="y",  # name of the value column
     freq="D"  # "H" for hourly, "D" for daily, "W" for weekly, etc.
 )

 forecaster = Forecaster()
 result = forecaster.run_forecast_config(
     df=df,
     config=ForecastConfig(
         model_template=ModelTemplateEnum.SILVERKITE.name,
         forecast_horizon=365,  # forecasts 365 steps ahead
         coverage=0.95,  # 95% prediction intervals
         metadata_param=metadata
     )
 )

Out:

Fitting 3 folds for each of 1 candidates, totalling 3 fits

For a detailed documentation about the output from run_forecast_config, see Check Forecast Result. Here we could plot the forecast.

 forecast = result.forecast
 fig = forecast.plot()
 plotly.io.show(fig)

Model performance evaluation

We can see the forecast fits the existing data well; however, we do not have a good ground truth to assess how well it predicts into the future.

Train-test-split

The typical way to evaluate model performance is to reserve part of the training data and use it to measure the model performance. Because we always predict the future in a time series forecasting problem, we reserve data from the end of training set to measure the performance of our forecasts. This is called a time series train test split.

By default, the results returned by run_forecast_config creates a time series train test split and stores the test result in result.backtest. The reserved testing data by default has the same length as the forecast horizon. We can access the evaluation results:

 pd.DataFrame(result.backtest.test_evaluation, index=["Value"]).transpose()  # formats dictionary as a pd.DataFrame

	Value
CORR	0.754144
R2	0.505377
MSE	0.252417
RMSE	0.502412
MAE	0.353357
MedAE	0.254763
MAPE	4.52689
MedAPE	3.27827
sMAPE	2.22671
Q80	0.125712
Q95	0.100229
Q99	0.0934336
OutsideTolerance1p	0.818182
OutsideTolerance2p	0.680441
OutsideTolerance3p	0.545455
OutsideTolerance4p	0.418733
OutsideTolerance5p	0.336088
Outside Tolerance (fraction)	None
R2_null_model_score	None
Prediction Band Width (%)	25.7568
Prediction Band Coverage (fraction)	0.983471
Coverage: Lower Band	0.735537
Coverage: Upper Band	0.247934
Coverage Diff: Actual_Coverage - Intended_Coverage	0.0334711

Evaluation metrics

From here we can see a list of metrics that measure the model performance on the test data. You may choose one or a few metrics to focus on. Typical metrics include:

MSE:

Mean squared error, the average squared error. Could be affected by extreme values.

RMSE:

Root mean squared error, the square root of MSE.

MAE:

Mean absolute error, the average of absolute error. Could be affected by extreme values.

MedAE:

Median absolute error, the median of absolute error. Less affected by extreme values.

MAPE:

Mean absolute percent error, measures the error percent with respective to the true values. This is useful when you would like to consider the relative error instead of the absolute error. For example, an error of 1 is considered as 10% for a true observation of 10, but as 1% for a true observation of 100. This is the default metric we like.

MedAPE:

Median absolute percent error, the median version of MAPE, less affected by extreme values.

Let’s use MAPE as our metric in this example. Looking at these results, you may have a basic sense of how the model is performing on the unseen test data. On average, the baseline model’s prediction is 11.3% away from the true values.

Time series cross-validation

Forecast quality depends a lot of the evaluation time window. The evaluation window selected above might happen to be a relatively easy/hard period to predict. Thus, it is more robust to evaluate over a longer time window when dataset size allows. Let’s consider a more general way of evaluating a forecast model: time series cross-validation.

Time series cross-validation is based on a time series rolling split. Let’s say we would like to perform an evaluation with a 3-fold cross-validation, The whole training data is split in 3 different ways. Since our forecast horizon is 365 days, we do:

First fold:

Train from 2007-12-10 to 2013-01-20, forecast from 2013-01-21 to 2014-01-20, and compare the forecast with the actual.

Second fold:

Train from 2007-12-10 to 2014-01-20, forecast from 2014-01-21 to 2015-01-20, and compare the forecast with the actual.

Third fold:

Train from 2007-12-10 to 2015-01-20, forecast from 2015-01-21 to 2016-01-20, and compare the forecast with the actual.

The split could be more flexible, for example, the testing periods could have gaps. For more details about evaluation period configuration, see Evaluation Period. The forecast model’s performance will be the average of the three evaluations on the forecasts.

By default, the results returned by run_forecast_config also runs time series cross-validation internally. You are allowed to configure the cross-validation splits, as shown below. Here note that the test_horizon are reserved from the back of the data and not used for cross-validation. This part of testing data can further evaluate the model performance besides the cross-validation result, and is available for plotting.

 # Defines the cross-validation config
 evaluation_period = EvaluationPeriodParam(
     test_horizon=365,             # leaves 365 days as testing data
     cv_horizon=365,               # each cv test size is 365 days (same as forecast horizon)
     cv_max_splits=3,              # 3 folds cv
     cv_min_train_periods=365 * 4  # uses at least 4 years for training because we have 8 years data
 )

 # Runs the forecast
 result = forecaster.run_forecast_config(
     df=df,
     config=ForecastConfig(
         model_template=ModelTemplateEnum.SILVERKITE.name,
         forecast_horizon=365,  # forecasts 365 steps ahead
         coverage=0.95,  # 95% prediction intervals
         metadata_param=metadata,
         evaluation_period_param=evaluation_period
     )
 )

 # Summarizes the cv result
 cv_results = summarize_grid_search_results(
     grid_search=result.grid_search,
     decimals=1,
     # The below saves space in the printed output. Remove to show all available metrics and columns.
     cv_report_metrics=None,
     column_order=["rank", "mean_test", "split_test", "mean_train", "split_train", "mean_fit_time", "mean_score_time", "params"])
 # Transposes to save space in the printed output
 cv_results["params"] = cv_results["params"].astype(str)
 cv_results.set_index("params", drop=True, inplace=True)
 cv_results.transpose()

Out:

Fitting 3 folds for each of 1 candidates, totalling 3 fits

params	[]
rank_test_MAPE	1
mean_test_MAPE	6.5
split_test_MAPE	(5.0, 6.9, 7.4)
mean_train_MAPE	4
split_train_MAPE	(3.9, 4.1, 4.0)
mean_fit_time	5.2
mean_score_time	0.9

By default, all metrics in ElementwiseEvaluationMetricEnum are computed on each CV train/test split. The configuration of CV evaluation metrics can be found at Evaluation Metric. Here, we show the Mean Absolute Percentage Error (MAPE) across splits (see summarize_grid_search_results to control what to show and for details on the output columns). From the result, we see that the cross-validation mean_test_MAPE is 7.3%, which means the prediction is 7.3% away from the ground truth on average. We also see the 3 cv folds have split_test_MAPE 5.1%, 8.5% and 8.4%, respectively.

When we have different sets of model parameters, a good way to compare them is to run a time series cross-validation on each set of parameters, and pick the set of parameters that has the best cross-validated performance.

Start tuning

Now that you know how to evaluate model performance, let’s see if we can improve the model by tuning its parameters.

Anomaly

An anomaly is a deviation in the metric that is not expected to occur again in the future. Including anomaly points will lead the model to fit the anomaly as an intrinsic property of the time series, resulting in inaccurate forecasts. These anomalies could be identified through overlay plots, see Seasonality Plots.

 fig = ts.plot_quantiles_and_overlays(
     groupby_time_feature="month_dom",
     show_mean=True,
     show_quantiles=False,
     show_overlays=True,
     overlay_label_time_feature="year",
     overlay_style={"line": {"width": 1}, "opacity": 0.5},
     center_values=True,
     xlabel="day of year",
     ylabel=ts.original_value_col,
     title="yearly seasonality for each year (centered)",
 )
 plotly.io.show(fig)

From the yearly overlay plot above, we could see two big anomalies: one in March of 2012, and one in June of 2010. Other small anomalies could be identified as well, however, they have less influence. The SILVERKITE template currently supports masking anomaly points by supplying the anomaly_info as a dictionary. You could either assign adjusted values to them, or simply mask them as NA (in which case these dates will not be used in fitting). For a detailed introduction about the anomaly_info configuration, see Examine Input Data. Here we define an anomaly_df dataframe to mask them as NA, and wrap it into the anomaly_info dictionary.

 anomaly_df = pd.DataFrame({
     # start and end date are inclusive
     # each row is an anomaly interval
     cst.START_TIME_COL: ["2010-06-05", "2012-03-01"],  # inclusive
     cst.END_TIME_COL: ["2010-06-20", "2012-03-20"],  # inclusive
     cst.ADJUSTMENT_DELTA_COL: [np.nan, np.nan],  # mask as NA
 })
 # Creates anomaly_info dictionary.
 # This will be fed into the template.
 anomaly_info = {
     "value_col": "y",
     "anomaly_df": anomaly_df,
     "adjustment_delta_col": cst.ADJUSTMENT_DELTA_COL,
 }

Adding relevant features

Growth and trend

First we look at the growth and trend. Detailed growth configuration can be found at Growth. In these two features, we care less about the short-term fluctuations but rather long-term tendency. From the original plot we see there is no obvious growth pattern, thus we could use a linear growth to fit the model. On the other hand, there could be potential trend changepoints, at which time the linear growth changes its rate. Detailed changepoint configuration can be found at Changepoints. These points can be detected with the ChangepointDetector class. For a quickstart example, see Changepoint Detection. Here we explore the automatic changepoint detection. The parameters in this automatic changepoint detection is customized for this data set. We keep the yearly_seasonality_order the same as the model’s yearly seasonality order. The regularization_strength controls how many changepoints are detected. 0.5 is a good choice, while you may try other numbers such as 0.4 or 0.6 to see the difference. The resample_freq is set to 7 days, because we have a long training history, thus we should keep this relatively long (the intuition is that shorter changes will be ignored). We put 25 potential changepoints to be the candidates, because we do not expect too many changes. However, this could be higher. The yearly_seasonality_change_freq is set to 365 days, which means we refit the yearly seasonality every year, because it can be see from the time series plot that the yearly seasonality varies every year. The no_changepoint_distance_from_end is set to 365 days, which means we do not allow any changepoints at the last 365 days of training data. This avoids fitting the final trend with too little data. For long-term forecast, this is typically the same as the forecast horizon, while for short-term forecast, this could be a multiple of the forecast horizon.

 model = ChangepointDetector()
 res = model.find_trend_changepoints(
     df=df,  # data df
     time_col="ts",  # time column name
     value_col="y",  # value column name
     yearly_seasonality_order=10,  # yearly seasonality order, fit along with trend
     regularization_strength=0.5,  # between 0.0 and 1.0, greater values imply fewer changepoints, and 1.0 implies no changepoints
     resample_freq="7D",  # data aggregation frequency, eliminate small fluctuation/seasonality
     potential_changepoint_n=25,  # the number of potential changepoints
     yearly_seasonality_change_freq="365D",  # varying yearly seasonality for every year
     no_changepoint_distance_from_end="365D")  # the proportion of data from end where changepoints are not allowed
 fig = model.plot(
     observation=True,
     trend_estimate=False,
     trend_change=True,
     yearly_seasonality_estimate=False,
     adaptive_lasso_estimate=True,
     plot=False)
 plotly.io.show(fig)

From the plot we see the automatically detected trend changepoints. The results shows that the time series is generally increasing until 2012, then generally decreasing. One possible explanation is that 2011 is the last year Peyton Manning was at the Indianapolis Colts before joining the Denver Broncos. If we feed the trend changepoint detection parameter to the template, these trend changepoint features will be automatically included in the model.

 # The following specifies the growth and trend changepoint configurations.
 growth = {
     "growth_term": "linear"
 }
 changepoints = {
     "changepoints_dict": dict(
         method="auto",
         yearly_seasonality_order=10,
         regularization_strength=0.5,
         resample_freq="7D",
         potential_changepoint_n=25,
         yearly_seasonality_change_freq="365D",
         no_changepoint_distance_from_end="365D"
     )
 }

Seasonality

The next features we will look into are the seasonality features. Detailed seasonality configurations can be found at Seasonality. A detailed seasonality detection quickstart example on the same data set is available at Seasonality Plots. The conclusions about seasonality terms are:

• daily seasonality is not available (because frequency is daily);

• weekly and yearly patterns are evident (weekly will also interact with football season);

• monthly or quarterly seasonality is not evident.

Therefore, for pure seasonality terms, we include weekly and yearly seasonality. The seasonality orders are something to be tuned; here let’s take weekly seasonality order to be 5 and yearly seasonality order to be 10. For tuning info, see Seasonality.

 # Includes yearly seasonality with order 10 and weekly seasonality with order 5.
 # Set the other seasonality to False to disable them.
 yearly_seasonality_order = 10
 weekly_seasonality_order = 5
 seasonality = {
     "yearly_seasonality": yearly_seasonality_order,
     "quarterly_seasonality": False,
     "monthly_seasonality": False,
     "weekly_seasonality": weekly_seasonality_order,
     "daily_seasonality": False
 }

We will add the interaction between weekly seasonality and the football season later in this tutorial. The SILVERKITE template also supports seasonality changepoints. A seasonality changepoint is a time point after which the periodic effect behaves differently. For SILVERKITE, this means the Fourier series coefficients are allowed to change. We could decide to add this feature if cross-validation performance is poor and seasonality changepoints are detected in exploratory analysis. For details, see Changepoint Detection.

Holidays and events

Then let’s look at holidays and events. Detailed holiday and event configurations can be found at Holidays and Events. Ask yourself which holidays are likely to affect the time series’ values. We expect that major United States holidays may affect wikipedia pageviews, since most football fans are in the United States. Events such as superbowl could potentially increase the pageviews. Therefore, we add United States holidays and superbowls dates as custom events. Other important events that affect the time series can also be found through the yearly seasonality plots in Seasonality Plots.

 # Includes major holidays and the superbowl date.
 events = {
     # These holidays as well as their pre/post dates are modeled as individual events.
     "holidays_to_model_separately": SilverkiteHoliday.ALL_HOLIDAYS_IN_COUNTRIES,  # all holidays in "holiday_lookup_countries"
     "holiday_lookup_countries": ["UnitedStates"],  # only look up holidays in the United States
     "holiday_pre_num_days": 2,  # also mark the 2 days before a holiday as holiday
     "holiday_post_num_days": 2,  # also mark the 2 days after a holiday as holiday
     "daily_event_df_dict": {
         "superbowl": pd.DataFrame({
             "date": ["2008-02-03", "2009-02-01", "2010-02-07", "2011-02-06",
                      "2012-02-05", "2013-02-03", "2014-02-02", "2015-02-01", "2016-02-07"],  # dates must cover training and forecast period.
             "event_name": ["event"] * 9  # labels
         })
     },
 }

Autoregression

The autoregressive features are very useful in short-term forecasting, but could be risky to use in long-term forecasting. Detailed autoregression configurations can be found at Auto-regression.

Custom

Now we consider some custom features that could relate to the pageviews. The documentation for extra regressors can be found at Regressors. As mentioned in Seasonality Plots, we observe that the football season heavily affects the pageviews, therefore we need to use regressors to identify the football season. There are multiple ways to include this feature: adding indicator for the whole season; adding number of days till season start (end) and number of days since season start (end). The former puts a uniform effect over all in-season dates, while the latter quantify the on-ramp and down-ramp. If you are not sure which effect to include, it’s ok to include both effects. SILVERKITE has the option to use Ridge regression as the fit algorithm to avoid over-fitting too many features. Note that many datetime features could also be added to the model as features. SILVERKITE calculates some of these features, which can be added to extra_pred_cols as an arbitrary patsy expression. For a full list of such features, see build_time_features_df.

If a feature is not automatically created by SILVERKITE, we need to create it beforehand and append it to the data df. Here we create the “is_football_season” feature. Note that we also need to provide the customized column for the forecast horizon period as well. The way we do it is to first create the df with timestamps covering the forecast horizon. This can be done with the make_future_dataframe function within the UnivariateTimeSeries class. Then we create a new column of our customized regressor for this augmented df.

 # Makes augmented df with forecast horizon 365 days
 df_full = ts.make_future_dataframe(periods=365)
 # Builds "df_features" that contains datetime information of the "df"
 df_features = build_time_features_df(
     dt=df_full["ts"],
     conti_year_origin=convert_date_to_continuous_time(df_full["ts"][0])
 )

 # Roughly approximates the football season.
 # "woy" is short for "week of year", created above.
 # Football season is roughly the first 6 weeks and last 17 weeks in a year.
 is_football_season = (df_features["woy"] <= 6) | (df_features["woy"] >= 36)
 # Adds the new feature to the dataframe.
 df_full["is_football_season"] = is_football_season.astype(int).tolist()
 df_full.reset_index(drop=True, inplace=True)

 # Configures regressor column.
 regressors = {
     "regressor_cols": ["is_football_season"]
 }

Interactions

Finally, let’s consider what possible interactions are relevant to the forecast problem. Generally speaking, if a feature behaves differently on different values of another feature, these two features could have potential interaction effects. As in Seasonality Plots, the weekly seasonality is different through football season and non-football season, therefore, the multiplicative term is_football_season x weekly_seasonality is able to capture this pattern.

 fig = ts.plot_quantiles_and_overlays(
     groupby_time_feature="str_dow",
     show_mean=True,
     show_quantiles=False,
     show_overlays=True,
     center_values=True,
     overlay_label_time_feature="month",  # splits overlays by month
     overlay_style={"line": {"width": 1}, "opacity": 0.5},
     xlabel="day of week",
     ylabel=ts.original_value_col,
     title="weekly seasonality by month",
 )
 plotly.io.show(fig)

Now let’s create the interaction terms: interaction between is_football_season and weekly seasonality. The interaction terms between a feature and a seasonality feature can be created with the cols_interact function.

 football_week = cols_interact(
     static_col="is_football_season",
     fs_name=SilverkiteSeasonalityEnum.WEEKLY_SEASONALITY.value.name,
     fs_order=weekly_seasonality_order,
     fs_seas_name=SilverkiteSeasonalityEnum.WEEKLY_SEASONALITY.value.seas_names
 )

 extra_pred_cols = football_week

Moreover, the multiplicative term month x weekly_seasonality and the dow_woy features also account for the varying weekly seasonality through the year. One could added these features, too. Here we just leave them out. You may use cols_interact again to create the month x weekly_seasonality similar to is_football_season x weekly_seasonality. dow_woy is automatically calcuated by SILVERKITE, you may simply append the name to extra_pred_cols to include it in the model.

Putting things together

Now let’s put everything together and produce a new forecast. A detailed template documentation can be found at Configure a Forecast. We first configure the MetadataParam class. The MetadataParam class includes basic proporties of the time series itself.

 metadata = MetadataParam(
     time_col="ts",              # column name of timestamps in the time series df
     value_col="y",              # column name of the time series values
     freq="D",                   # data frequency, here we have daily data
     anomaly_info=anomaly_info,  # this is the anomaly information we defined above,
     train_end_date=datetime.datetime(2016, 1, 20)
 )

Next we define the ModelComponentsParam class based on the discussion on relevant features. The ModelComponentsParam include properties related to the model itself.

 model_components = ModelComponentsParam(
     seasonality=seasonality,
     growth=growth,
     events=events,
     changepoints=changepoints,
     autoregression=None,
     regressors=regressors,  # is_football_season defined above
     uncertainty={
         "uncertainty_dict": "auto",
     },
     custom={
         # What algorithm is used to learn the relationship between the time series and the features.
         # Regularized fitting algorithms are recommended to mitigate high correlations and over-fitting.
         # If you are not sure what algorithm to use, "ridge" is a good choice.
         "fit_algorithm_dict": {
             "fit_algorithm": "ridge",
         },
         "extra_pred_cols": extra_pred_cols  # the interaction between is_football_season and weekly seasonality defined above
     }
 )

Now let’s run the model with the new configuration. The evaluation config is kept the same as the previous case; this is important for a fair comparison of parameter sets.

 # Runs the forecast
 result = forecaster.run_forecast_config(
     df=df_full,
     config=ForecastConfig(
         model_template=ModelTemplateEnum.SILVERKITE.name,
         forecast_horizon=365,  # forecasts 365 steps ahead
         coverage=0.95,  # 95% prediction intervals
         metadata_param=metadata,
         model_components_param=model_components,
         evaluation_period_param=evaluation_period
     )
 )

 # Summarizes the cv result
 cv_results = summarize_grid_search_results(
     grid_search=result.grid_search,
     decimals=1,
     # The below saves space in the printed output. Remove to show all available metrics and columns.
     cv_report_metrics=None,
     column_order=["rank", "mean_test", "split_test", "mean_train", "split_train", "mean_fit_time", "mean_score_time", "params"])
 # Transposes to save space in the printed output
 cv_results["params"] = cv_results["params"].astype(str)
 cv_results.set_index("params", drop=True, inplace=True)
 cv_results.transpose()

Out:

Fitting 3 folds for each of 1 candidates, totalling 3 fits

params	[]
rank_test_MAPE	1
mean_test_MAPE	5.6
split_test_MAPE	(3.9, 8.7, 4.3)
mean_train_MAPE	3.4
split_train_MAPE	(3.4, 3.6, 3.3)
mean_fit_time	5.3
mean_score_time	1.3

Now we see that after analyzing the problem and adding appropriate features, the cross-validation test MAPE is 5.4%, which is improved compared with the baseline (7.3%). The 3 cv folds also have their MAPE reduced to 3.9%, 8.7% and 3.8%, respectively. The first and third fold improved significantly. With some investigation, we can see that the second fold did not improve because there is a trend changepoint right at the the start of its test period.

It would be hard to know this situation until we see it. In the cross-validation step, one way to avoid this is to set a different evaluation period. However, leaving this period also makes sense because it could happen again in the future. In the forecast period, we could monitor the forecast and actual, and re-train the model to adapt to the most recent pattern if we see a deviation. In the changepoints dictionary, tune regularization_strength or no_changepoint_distance_from_end accordingly, or add manually specified changepoints to the automatically detected ones. For details, see Changepoints.

We could also plot the forecast.

 forecast = result.forecast
 fig = forecast.plot()
 plotly.io.show(fig)

Check model summary

To further investigate the model mechanism, it’s also helpful to see the model summary. The ModelSummary module provides model results such as estimations, significance, p-values, confidence intervals, etc. that can help the user understand how the model works and what can be further improved.

The model summary is a class method of the estimator and can be used as follows.

 summary = result.model[-1].summary()  # -1 retrieves the estimator from the pipeline
 print(summary)

Out:

================================ Model Summary =================================

Number of observations: 2964,   Number of features: 287
Method: Ridge regression
Number of nonzero features: 287
Regularization parameter: 0.3039

Residuals:
         Min           1Q       Median           3Q          Max
      -2.118      -0.2301     -0.05035       0.1607          3.2

             Pred_col    Estimate Std. Err Pr(>)_boot sig. code                  95%CI
            Intercept       6.865   0.1003     <2e-16       ***         (6.658, 7.035)
 events_Christmas Day      -0.447   0.1476      0.008        **     (-0.6977, -0.1425)
  events_C...bserved)      -0.825   0.6944      0.290                 (-1.873, 0.2918)
  events_C...erved)-1     -0.5959   0.6745      0.458                 (-1.628, 0.6782)
  events_C...erved)-2      -0.491   0.6259      0.512                  (-1.531, 0.428)
  events_C...erved)+1     0.04095   0.1691      0.704                (-0.3153, 0.3784)
  events_C...erved)+2     0.09565   0.1277      0.380                (-0.1461, 0.3946)
  events_C...as Day-1     -0.1761   0.1835      0.306                (-0.5733, 0.1641)
  events_C...as Day-2    -0.08813   0.2913      0.758                (-0.7677, 0.3899)
  events_C...as Day+1     -0.2733   0.1756      0.070         .     (-0.5742, 0.04129)
  events_C...as Day+2     0.07737   0.1283      0.520                (-0.1481, 0.3559)
  events_Columbus Day     -0.1725   0.2026      0.374                 (-0.5559, 0.248)
  events_C...us Day-1      0.1023   0.1129      0.380                (-0.1238, 0.3018)
  events_C...us Day-2    -0.06791  0.09759      0.478                (-0.2664, 0.1003)
  events_C...us Day+1   0.0009885   0.1417      0.992                (-0.2705, 0.2541)
  events_C...us Day+2   -0.005608   0.1537      0.978                (-0.2906, 0.3029)
  events_I...ence Day    -0.04401   0.1097      0.692                (-0.2422, 0.1836)
  events_I...bserved)     -0.1716   0.1187      0.134               (-0.3892, 0.06439)
  events_I...erved)-1     -0.1786   0.1465      0.196                (-0.4471, 0.1168)
  events_I...erved)-2      -0.107    0.112      0.332                (-0.2819, 0.1389)
  events_I...erved)+1     0.02887   0.1495      0.860                 (-0.2323, 0.341)
  events_I...erved)+2      0.1476   0.2461      0.570                 (-0.1935, 0.707)
  events_I...ce Day-1     -0.1358  0.08685      0.100               (-0.2851, 0.04469)
  events_I...ce Day-2     -0.1182  0.09031      0.172               (-0.3146, 0.05781)
  events_I...ce Day+1     -0.0547   0.1078      0.634                (-0.2511, 0.1692)
  events_I...ce Day+2    -0.05212   0.1209      0.668                (-0.2455, 0.2048)
     events_Labor Day      -1.269   0.1362     <2e-16       ***       (-1.538, -1.009)
   events_Labor Day-1     -0.1169   0.1812      0.496                (-0.4768, 0.2288)
   events_Labor Day-2      -0.112  0.08055      0.154               (-0.2737, 0.04546)
   events_Labor Day+1     -0.6876    0.116      0.002        **      (-0.912, -0.4486)
   events_Labor Day+2     -0.2759   0.1109      0.012         *    (-0.4978, -0.07521)
  events_M... Jr. Day      0.3543   0.2873      0.218                (-0.1871, 0.9906)
  events_M...r. Day-1      0.3361   0.3105      0.290                (-0.2922, 0.8995)
  events_M...r. Day-2    -0.07002   0.1309      0.588                (-0.3264, 0.1756)
  events_M...r. Day+1     -0.1021    0.203      0.606                (-0.5186, 0.2678)
  events_M...r. Day+2     0.06881   0.1653      0.692                (-0.2412, 0.3915)
  events_Memorial Day      -0.228  0.05559     <2e-16       ***       (-0.327, -0.104)
  events_M...al Day-1     -0.1648  0.09766      0.078         .      (-0.3069, 0.0605)
  events_M...al Day-2    -0.09376   0.1217      0.434                (-0.2785, 0.1717)
  events_M...al Day+1     -0.0508  0.08201      0.520                (-0.2021, 0.1185)
  events_M...al Day+2       0.112   0.1067      0.296               (-0.09391, 0.3215)
 events_New Years Day     -0.2211   0.1039      0.034         *    (-0.4114, -0.01287)
  events_N...bserved)     0.01098  0.08275      0.754                (-0.1698, 0.1775)
  events_N...erved)-1     -0.2069   0.1301      0.082         .     (-0.4199, 0.05338)
  events_N...erved)-2     -0.1311   0.3337      0.764                (-0.7334, 0.3636)
  events_N...erved)+1      0.3919   0.2724      0.104               (-0.09078, 0.7901)
  events_N...erved)+2     0.07463  0.08033      0.314               (-0.08184, 0.2189)
  events_N...rs Day-1    -0.03627   0.1145      0.720                (-0.2394, 0.1904)
  events_N...rs Day-2      0.1363   0.1695      0.414                (-0.1893, 0.4777)
  events_N...rs Day+1      0.1952   0.1056      0.058         .     (-0.001367, 0.402)
  events_N...rs Day+2      0.2089   0.1638      0.204                (-0.1001, 0.5602)
  events_Thanksgiving     -0.1123   0.1025      0.268               (-0.3017, 0.09227)
  events_T...giving-1     -0.3524   0.0903     <2e-16       ***     (-0.5176, -0.1696)
  events_T...giving-2     -0.3579   0.1019      0.002        **     (-0.5566, -0.1474)
  events_T...giving+1     -0.0696   0.1282      0.576                (-0.2992, 0.2156)
  events_T...giving+2     -0.1648  0.08458      0.052         .   (-0.3495, -0.002858)
  events_Veterans Day    -0.07666  0.09382      0.428                (-0.2577, 0.1121)
  events_V...bserved)     -0.3612   0.1983      0.006        **          (-0.5328, 0.)
  events_V...erved)-1       0.145  0.09903      0.104               (-0.03869, 0.3001)
  events_V...erved)-2     0.01196  0.05362      0.540                (-0.1071, 0.1343)
  events_V...erved)+1       -0.17   0.1015      0.048         *          (-0.2981, 0.)
  events_V...erved)+2     -0.1681  0.09446      0.022         *          (-0.2709, 0.)
  events_V...ns Day-1     -0.1217  0.07352      0.092         .     (-0.2602, 0.02886)
  events_V...ns Day-2    -0.03078  0.09859      0.756                (-0.2081, 0.1703)
  events_V...ns Day+1    -0.01522   0.0993      0.872                (-0.2203, 0.1734)
  events_V...ns Day+2    -0.06372  0.06513      0.326               (-0.1781, 0.06669)
  events_W...Birthday    -0.03229   0.1231      0.792                (-0.2752, 0.2289)
  events_W...rthday-1     -0.3397  0.08975      0.002        **      (-0.511, -0.1733)
  events_W...rthday-2     -0.1428  0.07618      0.052         .    (-0.2797, 0.006259)
  events_W...rthday+1    -0.07234  0.08953      0.428                (-0.2301, 0.1114)
  events_W...rthday+2     -0.1424  0.05091      0.004        **    (-0.2422, -0.04568)
     events_superbowl      0.5842   0.3422      0.088         .      (-0.05574, 1.267)
        str_dow_2-Tue     0.03967  0.02259      0.076         .   (-0.003986, 0.08442)
        str_dow_3-Wed     0.04804  0.01906      0.020         *       (0.012, 0.08583)
        str_dow_4-Thu     0.03556  0.02056      0.076         .   (-0.003295, 0.07609)
        str_dow_5-Fri    -0.02513  0.02013      0.202                (-0.064, 0.01469)
        str_dow_6-Sat    -0.09027  0.01865     <2e-16       ***    (-0.1259, -0.05314)
        str_dow_7-Sun    -0.03796   0.0254      0.134             (-0.08553, 0.008544)
  is_footb...w_weekly    -0.09077  0.04617      0.068         .    (-0.1897, 0.002181)
  is_footb...w_weekly      0.9158  0.05201     <2e-16       ***        (0.8126, 1.018)
  is_footb...w_weekly    -0.03435  0.02444      0.152               (-0.0816, 0.01376)
  is_footb...w_weekly       0.193  0.02458     <2e-16       ***       (0.1461, 0.2406)
  is_footb...w_weekly    -0.05498  0.02267      0.014         *   (-0.09745, -0.01005)
  is_footb...w_weekly      0.0214  0.02568      0.402                (-0.029, 0.07263)
  is_footb...w_weekly     0.05498  0.02267      0.014         *     (0.01005, 0.09745)
  is_footb...w_weekly      0.0214  0.02568      0.402                (-0.029, 0.07263)
  is_footb...w_weekly     0.03435  0.02444      0.152               (-0.01376, 0.0816)
  is_footb...w_weekly       0.193  0.02458     <2e-16       ***       (0.1461, 0.2406)
   is_football_season      0.5986   0.1072     <2e-16       ***       (0.4135, 0.8328)
                  ct1      -2.116    0.273     <2e-16       ***       (-2.626, -1.561)
       is_weekend:ct1     -0.4368   0.1825      0.016         *    (-0.7881, -0.07116)
    str_dow_2-Tue:ct1     -0.2786   0.2211      0.218                 (-0.6943, 0.168)
    str_dow_3-Wed:ct1     -0.2534   0.1449      0.084         .     (-0.5598, 0.02246)
    str_dow_4-Thu:ct1     -0.2471   0.1383      0.074         .  (-0.5298, -4.691e-05)
    str_dow_5-Fri:ct1    -0.02971   0.1359      0.796                (-0.3123, 0.2409)
    str_dow_6-Sat:ct1    -0.07955   0.1093      0.454                 (-0.292, 0.1292)
    str_dow_7-Sun:ct1     -0.3573   0.2167      0.094         .     (-0.7796, 0.07526)
    cp0_2008_07_21_00      0.1131   0.1287      0.386               (-0.09206, 0.3842)
  is_weeke...07_21_00   -0.005642  0.08358      0.942                (-0.1749, 0.1457)
  str_dow_...07_21_00     0.01441   0.0982      0.898                (-0.1422, 0.2192)
  str_dow_...07_21_00     0.05431  0.06842      0.422               (-0.07867, 0.2095)
  str_dow_...07_21_00     -0.0155  0.07517      0.854                 (-0.1547, 0.135)
  str_dow_...07_21_00     0.08226  0.08382      0.324               (-0.08008, 0.2412)
  str_dow_...07_21_00    -0.02529  0.07451      0.722                (-0.1674, 0.1175)
  str_dow_...07_21_00     0.01964  0.08095      0.818                (-0.1329, 0.1775)
    cp1_2008_11_10_00       3.173   0.1849     <2e-16       ***         (2.695, 3.466)
  is_weeke...11_10_00      0.7364   0.1214     <2e-16       ***        (0.472, 0.9505)
  str_dow_...11_10_00      0.4921    0.125     <2e-16       ***       (0.2415, 0.7369)
  str_dow_...11_10_00      0.4083  0.09213     <2e-16       ***       (0.2267, 0.5686)
  str_dow_...11_10_00      0.3471  0.08062     <2e-16       ***       (0.2089, 0.5067)
  str_dow_...11_10_00      0.4143   0.1091     <2e-16       ***        (0.227, 0.6345)
  str_dow_...11_10_00      0.3338  0.09334     <2e-16       ***       (0.1471, 0.5234)
  str_dow_...11_10_00      0.4028   0.1254      0.004        **       (0.1383, 0.6278)
    cp2_2009_03_09_00       2.362   0.2022     <2e-16       ***          (1.922, 2.72)
  is_weeke...03_09_00      0.6321   0.1498     <2e-16       ***       (0.3421, 0.9158)
  str_dow_...03_09_00      0.3632   0.1458      0.014         *      (0.06062, 0.6302)
  str_dow_...03_09_00      0.2201   0.1115      0.050         .    (-0.003051, 0.4374)
  str_dow_...03_09_00      0.2257  0.09796      0.022         *      (0.03279, 0.4128)
  str_dow_...03_09_00      0.2769   0.1206      0.030         *        (0.057, 0.5457)
  str_dow_...03_09_00      0.2989    0.116      0.010         *      (0.07465, 0.5216)
  str_dow_...03_09_00      0.3333   0.1614      0.038         *        (0.015, 0.6407)
    cp3_2009_10_19_00      -1.466   0.2267     <2e-16       ***      (-1.912, -0.9744)
  is_weeke...10_19_00     -0.4341   0.1377     <2e-16       ***     (-0.7007, -0.1423)
  str_dow_...10_19_00     -0.2765   0.1722      0.094         .      (-0.614, 0.03005)
  str_dow_...10_19_00     -0.2843   0.1234      0.014         *    (-0.5221, -0.04596)
  str_dow_...10_19_00     -0.1583   0.1213      0.204                (-0.3853, 0.0756)
  str_dow_...10_19_00     -0.2632   0.1494      0.094         .     (-0.5511, 0.04918)
  str_dow_...10_19_00     -0.2036   0.1218      0.096         .       (-0.45, 0.04807)
  str_dow_...10_19_00     -0.2305   0.1442      0.112                (-0.484, 0.07072)
    cp4_2010_02_15_00      -1.794    0.224     <2e-16       ***       (-2.205, -1.313)
  is_weeke...02_15_00     -0.5687   0.1705      0.004        **     (-0.8824, -0.2342)
  str_dow_...02_15_00     -0.3043   0.1709      0.080         .     (-0.6524, 0.01207)
  str_dow_...02_15_00     -0.2483   0.1407      0.080         .     (-0.5359, 0.02631)
  str_dow_...02_15_00     -0.1491   0.1281      0.240               (-0.4231, 0.09173)
  str_dow_...02_15_00     -0.3762   0.1782      0.038         *    (-0.7325, -0.03526)
  str_dow_...02_15_00     -0.2897   0.1408      0.032         *    (-0.5435, -0.00825)
  str_dow_...02_15_00     -0.2791    0.177      0.106               (-0.6075, 0.05548)
    cp5_2010_06_07_00     -0.1806   0.1826      0.310                (-0.6086, 0.1308)
  is_weeke...06_07_00    -0.07667    0.133      0.578                  (-0.34, 0.1783)
  str_dow_...06_07_00     -0.1238   0.1406      0.398                (-0.4234, 0.1265)
  str_dow_...06_07_00      0.1082   0.1053      0.322                (-0.0988, 0.3036)
  str_dow_...06_07_00     0.08429   0.1026      0.388                (-0.1152, 0.2856)
  str_dow_...06_07_00     -0.1798   0.1446      0.214               (-0.4822, 0.08929)
  str_dow_...06_07_00    -0.03911   0.1214      0.732                (-0.2689, 0.1968)
  str_dow_...06_07_00    -0.03758   0.1425      0.758                (-0.3015, 0.2469)
    cp6_2011_01_24_00       1.252   0.2672     <2e-16       ***         (0.802, 1.813)
  is_weeke...01_24_00      0.3132   0.2037      0.114               (-0.09599, 0.7063)
  str_dow_...01_24_00      0.1669    0.217      0.450                (-0.2546, 0.5955)
  str_dow_...01_24_00      0.1903   0.1496      0.188                 (-0.09922, 0.52)
  str_dow_...01_24_00      0.1045   0.1483      0.452                (-0.1772, 0.4391)
  str_dow_...01_24_00     0.07014   0.1919      0.714                (-0.2602, 0.4578)
  str_dow_...01_24_00       0.109   0.1544      0.456                (-0.1898, 0.4417)
  str_dow_...01_24_00      0.2043   0.2418      0.402                (-0.3027, 0.6531)
    cp7_2011_05_16_00       1.531     0.22     <2e-16       ***         (1.122, 2.006)
  is_weeke...05_16_00      0.3425   0.1759      0.050         .      (0.03776, 0.7171)
  str_dow_...05_16_00      0.2928   0.1555      0.064         .     (0.006858, 0.6215)
  str_dow_...05_16_00       0.216   0.1547      0.172               (-0.08739, 0.5504)
  str_dow_...05_16_00        0.13   0.1461      0.364                (-0.1587, 0.4101)
  str_dow_...05_16_00      0.2228   0.1832      0.236                (-0.1333, 0.5891)
  str_dow_...05_16_00         0.1   0.1785      0.554                (-0.2684, 0.4267)
  str_dow_...05_16_00      0.2425   0.1625      0.148               (-0.05758, 0.5547)
    cp8_2012_01_02_00     -0.8718   0.3015      0.004        **      (-1.621, -0.4352)
  is_weeke...01_02_00     0.02033   0.1785      0.908                (-0.3552, 0.2974)
  str_dow_...01_02_00     -0.1828   0.2172      0.426                (-0.6306, 0.2163)
  str_dow_...01_02_00     -0.3025   0.1746      0.070         .     (-0.6724, 0.01745)
  str_dow_...01_02_00     -0.2907   0.1634      0.074         .     (-0.6328, 0.03005)
  str_dow_...01_02_00      0.2025   0.2377      0.424                (-0.2787, 0.6238)
  str_dow_...01_02_00     0.00292   0.1765      0.988                (-0.3917, 0.2929)
  str_dow_...01_02_00     0.01742   0.1864      0.938                (-0.3466, 0.3544)
    cp9_2012_04_23_00      -2.969   0.2505     <2e-16       ***       (-3.395, -2.427)
  is_weeke...04_23_00     -0.5085   0.1939      0.008        **      (-0.879, -0.1548)
  str_dow_...04_23_00     -0.4973   0.1966      0.014         *     (-0.8721, -0.1339)
  str_dow_...04_23_00     -0.5037   0.1999      0.014         *     (-0.9676, -0.1515)
  str_dow_...04_23_00     -0.3741   0.1647      0.020         *    (-0.6951, -0.06403)
  str_dow_...04_23_00     -0.3986     0.19      0.034         *    (-0.7843, -0.03672)
  str_dow_...04_23_00     -0.2649   0.1764      0.132                (-0.624, 0.06608)
  str_dow_...04_23_00     -0.2438   0.1861      0.190                 (-0.635, 0.1162)
   cp10_2012_08_13_00     -0.2964    0.269      0.272                (-0.8085, 0.1963)
  is_weeke...08_13_00     -0.3269   0.1511      0.030         *    (-0.6216, -0.02609)
  str_dow_...08_13_00    -0.03866   0.1966      0.842                (-0.4129, 0.3409)
  str_dow_...08_13_00       0.261   0.1268      0.038         *     (0.001913, 0.4881)
  str_dow_...08_13_00      0.1979   0.1553      0.202                (-0.1419, 0.4631)
  str_dow_...08_13_00     -0.1781   0.1572      0.248                 (-0.4656, 0.133)
  str_dow_...08_13_00     0.01131   0.1205      0.924                (-0.1989, 0.2775)
  str_dow_...08_13_00     -0.3382   0.1595      0.046         *    (-0.6799, -0.03916)
   cp11_2013_04_01_00       1.875   0.1795     <2e-16       ***         (1.547, 2.219)
  is_weeke...04_01_00      0.2317   0.1763      0.190                 (-0.106, 0.5898)
  str_dow_...04_01_00      0.3977   0.2071      0.050         .      (0.01939, 0.8174)
  str_dow_...04_01_00      0.2649   0.1783      0.130                 (-0.0661, 0.601)
  str_dow_...04_01_00      0.2389   0.1727      0.176               (-0.09585, 0.5648)
  str_dow_...04_01_00      0.2261   0.1942      0.226                 (-0.1532, 0.609)
  str_dow_...04_01_00     0.09089    0.184      0.648                 (-0.258, 0.4518)
  str_dow_...04_01_00      0.1409   0.2386      0.548                 (-0.304, 0.6097)
   cp12_2014_03_10_00     -0.9337   0.1466     <2e-16       ***      (-1.211, -0.6377)
  is_weeke...03_10_00    -0.07739   0.1179      0.526                (-0.3195, 0.1474)
  str_dow_...03_10_00   -0.002764   0.2224      0.998                (-0.4147, 0.4099)
  str_dow_...03_10_00     -0.1758   0.1239      0.154               (-0.4118, 0.05836)
  str_dow_...03_10_00    -0.09518    0.145      0.518                 (-0.388, 0.1747)
  str_dow_...03_10_00    -0.02337   0.1595      0.906                (-0.3239, 0.2835)
  str_dow_...03_10_00     -0.1103   0.1394      0.420                (-0.3911, 0.1686)
  str_dow_...03_10_00     0.03272   0.1948      0.864                (-0.3365, 0.4093)
  ct1:sin1_tow_weekly     -0.1039   0.1374      0.478                (-0.3526, 0.1567)
  ct1:cos1_tow_weekly     -0.4974   0.2486      0.046         *     (-0.9236, 0.05699)
  ct1:sin2_tow_weekly       0.165    0.161      0.314                (-0.1755, 0.4563)
  ct1:cos2_tow_weekly     -0.3169   0.2378      0.172                (-0.8066, 0.1262)
  cp0_2008...w_weekly     0.01599  0.06061      0.788               (-0.09778, 0.1441)
  cp0_2008...w_weekly    -0.03264   0.1012      0.720                (-0.2209, 0.1889)
  cp0_2008...w_weekly     0.01883  0.06642      0.774                (-0.1054, 0.1501)
  cp0_2008...w_weekly   -0.004663  0.09841      0.966                 (-0.194, 0.1882)
  cp1_2008...w_weekly     0.05856  0.08946      0.536                (-0.1106, 0.2307)
  cp1_2008...w_weekly      0.2559    0.133      0.060         .      (-0.02328, 0.496)
  cp1_2008...w_weekly     0.05508  0.09863      0.590                (-0.1236, 0.2529)
  cp1_2008...w_weekly      0.2023   0.1276      0.112                (-0.0504, 0.4271)
  cp2_2009...w_weekly    -0.03844   0.1055      0.722                (-0.2391, 0.1578)
  cp2_2009...w_weekly        0.27   0.1469      0.064         .     (-0.01899, 0.5562)
  cp2_2009...w_weekly     0.05305   0.1244      0.658                (-0.1961, 0.2818)
  cp2_2009...w_weekly      0.1771   0.1413      0.212               (-0.09249, 0.4431)
  cp3_2009...w_weekly    -0.03576    0.113      0.766                (-0.2619, 0.1718)
  cp3_2009...w_weekly      0.0632   0.1815      0.718                (-0.2816, 0.4167)
  cp3_2009...w_weekly    -0.04707   0.1292      0.684                (-0.3294, 0.1907)
  cp3_2009...w_weekly      0.1254   0.1604      0.414                (-0.1904, 0.4455)
  cp4_2010...w_weekly     0.06075   0.1254      0.640                 (-0.1801, 0.315)
  cp4_2010...w_weekly     0.04123   0.1796      0.802                (-0.3318, 0.3906)
  cp4_2010...w_weekly     -0.1128    0.149      0.482                (-0.3862, 0.1776)
  cp4_2010...w_weekly     0.07216   0.1637      0.690                (-0.2424, 0.3758)
  cp5_2010...w_weekly     0.09778  0.09842      0.322                 (-0.119, 0.2803)
  cp5_2010...w_weekly    -0.01226     0.14      0.942                (-0.2914, 0.2553)
  cp5_2010...w_weekly     -0.1817   0.1174      0.132               (-0.4065, 0.05105)
  cp5_2010...w_weekly    -0.04164   0.1265      0.770                (-0.2868, 0.1951)
  cp6_2011...w_weekly     0.03351   0.1454      0.806                (-0.2483, 0.3223)
  cp6_2011...w_weekly      0.2194   0.2013      0.264                (-0.2023, 0.5799)
  cp6_2011...w_weekly    -0.05048   0.1863      0.768                (-0.4022, 0.3122)
  cp6_2011...w_weekly     0.08697   0.1793      0.618                (-0.3058, 0.4405)
  cp7_2011...w_weekly     0.05783   0.1406      0.678                (-0.2085, 0.3561)
  cp7_2011...w_weekly      0.1453   0.1573      0.350                (-0.1665, 0.4543)
  cp7_2011...w_weekly     0.03656   0.1411      0.792                (-0.2317, 0.2908)
  cp7_2011...w_weekly     0.07633   0.1606      0.620                (-0.2504, 0.3655)
  cp8_2012...w_weekly     -0.3427   0.1564      0.020         *   (-0.6028, -0.009657)
  cp8_2012...w_weekly     -0.1459   0.2379      0.532                (-0.6294, 0.3112)
  cp8_2012...w_weekly      0.1652   0.1803      0.346                (-0.2279, 0.4883)
  cp8_2012...w_weekly    -0.03538    0.221      0.898                (-0.4275, 0.3752)
  cp9_2012...w_weekly     -0.2163    0.163      0.182                (-0.5389, 0.1065)
  cp9_2012...w_weekly     -0.1521   0.1916      0.420                (-0.5377, 0.2132)
  cp9_2012...w_weekly    -0.08354   0.1587      0.624                (-0.3971, 0.2143)
  cp9_2012...w_weekly     -0.1656   0.1887      0.368                (-0.5404, 0.1865)
  cp10_201...w_weekly      0.3285   0.1254      0.008        **      (0.06736, 0.5487)
  cp10_201...w_weekly     -0.2768   0.1951      0.138                (-0.6431, 0.1254)
  cp10_201...w_weekly    -0.05635   0.1513      0.708                (-0.3427, 0.2502)
  cp10_201...w_weekly     -0.1903   0.1748      0.274                  (-0.54, 0.1224)
  cp11_201...w_weekly      0.1933   0.1459      0.196                 (-0.0901, 0.482)
  cp11_201...w_weekly      0.1893   0.2198      0.382                (-0.2601, 0.6006)
  cp11_201...w_weekly     0.08475   0.1717      0.606                (-0.2561, 0.4209)
  cp11_201...w_weekly      0.1936   0.1873      0.316                (-0.1745, 0.5771)
  cp12_201...w_weekly    -0.06325  0.09738      0.510                (-0.2653, 0.1155)
  cp12_201...w_weekly      -0.197   0.2423      0.408                (-0.6995, 0.2455)
  cp12_201...w_weekly     0.02565   0.1235      0.844                (-0.2034, 0.2709)
  cp12_201...w_weekly      -0.202   0.2167      0.362                (-0.6141, 0.2004)
      sin1_tow_weekly      0.1138  0.02859     <2e-16       ***      (0.05877, 0.1723)
      cos1_tow_weekly     0.01639  0.03983      0.690              (-0.06546, 0.08612)
      sin2_tow_weekly     -0.0163  0.01923      0.416              (-0.05149, 0.02123)
      cos2_tow_weekly     0.03907  0.02623      0.126              (-0.01579, 0.08689)
      sin3_tow_weekly   -0.007843  0.01401      0.560              (-0.03469, 0.01949)
      cos3_tow_weekly  -5.704e-05  0.02327      0.998              (-0.04146, 0.04806)
      sin4_tow_weekly    0.007843  0.01401      0.560              (-0.01949, 0.03469)
      cos4_tow_weekly  -5.704e-05  0.02327      0.998              (-0.04146, 0.04806)
      sin5_tow_weekly      0.0163  0.01923      0.416              (-0.02123, 0.05149)
      cos5_tow_weekly     0.03907  0.02623      0.126              (-0.01579, 0.08689)
      sin1_ct1_yearly    -0.05494  0.04838      0.246               (-0.1427, 0.04508)
      cos1_ct1_yearly      0.7379   0.1288     <2e-16       ***       (0.4429, 0.9576)
      sin2_ct1_yearly      0.2218  0.03018     <2e-16       ***       (0.1616, 0.2846)
      cos2_ct1_yearly     -0.3146  0.03404     <2e-16       ***     (-0.3902, -0.2612)
      sin3_ct1_yearly      0.3322  0.03773     <2e-16       ***       (0.2591, 0.4078)
      cos3_ct1_yearly    0.004293  0.03136      0.876              (-0.05236, 0.07283)
      sin4_ct1_yearly     -0.1058   0.0359      0.002        **    (-0.1791, -0.03255)
      cos4_ct1_yearly       -0.16  0.02612     <2e-16       ***      (-0.207, -0.1027)
      sin5_ct1_yearly     -0.1171  0.02792     <2e-16       ***    (-0.1706, -0.05889)
      cos5_ct1_yearly    -0.03745  0.02197      0.100             (-0.07969, 0.008997)
      sin6_ct1_yearly     -0.1119  0.03282     <2e-16       ***    (-0.1755, -0.04998)
      cos6_ct1_yearly    -0.02081   0.0246      0.408              (-0.07106, 0.02497)
      sin7_ct1_yearly    -0.09032  0.02398      0.002        **    (-0.1338, -0.04659)
      cos7_ct1_yearly     0.05681  0.02381      0.018         *     (0.008143, 0.1052)
      sin8_ct1_yearly     0.02703  0.02671      0.300               (-0.03017, 0.0736)
      cos8_ct1_yearly      0.1241  0.02765     <2e-16       ***       (0.06811, 0.176)
      sin9_ct1_yearly    -0.01225  0.02377      0.604              (-0.06104, 0.03254)
      cos9_ct1_yearly    -0.07556  0.02507      0.002        **    (-0.1306, -0.03028)
     sin10_ct1_yearly      -0.126  0.02326     <2e-16       ***    (-0.1725, -0.08198)
     cos10_ct1_yearly    -0.04909  0.02404      0.038         *    (-0.08949, 0.00236)
Signif. Code: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Multiple R-squared: 0.7512,   Adjusted R-squared: 0.7389
F-statistic: 59.643 on 139 and 2823 DF,   p-value: 1.110e-16
Model AIC: 18788.0,   model BIC: 19630.0

WARNING: the condition number is large, 2.60e+05. This might indicate that there are strong multicollinearity or other numerical problems.
WARNING: the F-ratio and its p-value on regularized methods might be misleading, they are provided only for reference purposes.

The model summary shows the model information, the coefficients and their significance, and a few summary statistics. For example, we can see the changepoints and how much the growth rate changes at each changepoint. We can see that some of the holidays have significant effect in the model, such as Christmas, Labor day, Thanksgiving, etc. We can see the significance of the interaction between football season and weekly seasonality etc.

For a more detailed guide on model summary, see Model Summary.

Summary in model tuning

After the example, you may have some sense about how to select parameters and tune the model. Here we list a few steps and tricks that might help select the best models. What you may do:

1. Detect anomaly points with the overlay plots (plot_quantiles_and_overlays). Mask these points with NA. Do not specify the adjustment unless you are confident about how to correct the anomalies.

2. Choose an appropriate way to model the growth (linear, quadratic, square root, etc.) If none of the typical growth shape fits the time series, you might consider linear growth with trend changepoints. Try different changepoint detection configurations. You may also plot the detected changepoints and see if it makes sense to you. The template also supports custom changepoints. If the automatic changepoint detection result does not make sense to you, you might supply your own changepoints.

3. Choose the appropriate seasonality orders. The higher the order, the more details the model can learn. However, too large orders could overfit the training data. These can also be detected from the overlay plots (plot_quantiles_and_overlays). There isn’t a unified way to choose seasonality, so explore different seasonality orders and compare the results.

4. Consider what events and holidays to model. Are there any custom events we need to add? If you add a custom event, remember also adding the dates for the event in the forecast period.

5. Add external regressors that could be related to the time series. Note that you will need to provide the values of the regressors in the forecast period as well. You may use another time series as a regressor, as long as you have a ground truth/good forecast for it that covers your forecast period.

6. Adding interaction terms. Let’s mention again here that there could be interaction between two features if the behaviors of one feature are different when the other feature have different values. Try to detect this through the overlay plot (plot_quantiles_and_overlays), too. By default, we have a few pre-defined interaction terms, see feature_sets_enabled.

7. Choose an appropriate fit algorithm. This is the algorithm that models the relationship between the features and the time series. See a full list of available algorithms at fit_algorithm. If you are unsure about their difference, try some of them and compare the results. If you don’t want to, choosing “ridge” is a safe option.

It is worth noting that the template supports automatic grid search with different sets of parameters. For each parameter, if you provide the configuration in a list, it will automatically run each combination and choose the one with the best cross-validation performance. This will save a lot of time. For details, see Grid Search.

Follow your insights and intuitions, and play with the parameters, you will get good forecasts!