Note
Click here to download the full example code
Tune your first forecast model¶
This is a basic tutorial for creating and tuning a forecast model. It is intended to provide a basic sense of a forecast process without assuming background knowledge in forecasting.
You can use the PROPHET or SILVERKITE model.
In this tutorial, we focus on SILVERKITE.
However, the basic ideas of tuning are similar to both models.
You may see detailed information about PROPHET at
Prophet.
SILVERKITE decomposes time series into various components, and it
creates time-based features, autoregressive features,
together with user-provided features such as macro-economic features
and their interactions, then performs a machine learning regression
model to learn the relationship between the time series and these
features. The forecast is based on the learned relationship and
the future values of these features. Therefore, including the correct
features is the key to success.
Common features include:
- Datetime derivatives:
Including features derived from datetime such as
day of year,hour of day,weekday,is_weekendand etc. These features are useful in capturing special patterns. For example, the patterns of weekdays and weekends are different for most business related time series, and this can be modeled withis_weekend.- Growth:
First defines the basic feature
ct1that counts how long has passed in terms of years (could be fraction) since the first day of training data. For example, if the training data starts with “2018-01-01”, then the date hasct1=0.0, and “2018-01-02” hasct1=1/365. “2019-01-01” hasct1=1.0. Thisct1can be as granular as needed. A separate growth function can be applied toct1to support different types of growth model. For example,ct2is defined as the square ofct1to model quadratic growth.- Trend:
Trend describes the average tendency of the time series. It is defined through the growth term with possible changepoints. At every changepoint, the growth rate could change (faster or slower). For example, if
ct1(linear growth) is used with changepoints, the trend is modeled as piece-wise linear.- Seasonality:
Seasonality describes the periodical pattern of the time series. It contains multiple levels including daily seasonality, weekly seasonality, monthly seasonality, quarterly seasonality and yearly seasonality. Seasonality are defined through Fourier series with different orders. The greater the order, the more detailed periodical pattern the model can learn. However, an order that is too large can lead to overfitting.
- Events:
Events include holidays and other short-term occurrences that could temporarily affect the time series, such as Thanksgiving long weekend. Typically, events are regular and repeat at know times in the future. These features made of indicators that covers the event day and their neighbor days.
- Autoregression:
Autoregressive features include the time series observations in the past and their aggregations. For example, the past day’s observation, the same weekday on the past week, or the average of the past 7 days, etc. can be used. Note that autoregression features are very useful in short term forecasts, however, this should be avoided in long term forecast. The reason is that long-term forecast focuses more on the correctness of trend, seasonality and events. The lags and autoregressive terms in a long-term forecast are calculated based on the forecasted values. The further we forecast into the future, the more forecasted values we need to create the autoregressive terms, making the forecast less stable.
- Custom:
Extra features that are relevant to the time series such as macro-ecomonic features that are expected to affect the time series. Note that these features need to be manually provided for both the training and forecasting periods.
- Interactions:
Any interaction between the features above.
Now let’s use an example to go through the full forecasting and tuning process.
In this example, we’ll load a dataset representing log(daily page views)
on the Wikipedia page for Peyton Manning.
It contains values from 2007-12-10 to 2016-01-20. More dataset info
here.
87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 | import datetime
import numpy as np
import pandas as pd
import plotly
from greykite.algo.changepoint.adalasso.changepoint_detector import ChangepointDetector
from greykite.algo.forecast.silverkite.constants.silverkite_holiday import SilverkiteHoliday
from greykite.algo.forecast.silverkite.constants.silverkite_seasonality import SilverkiteSeasonalityEnum
from greykite.algo.forecast.silverkite.forecast_simple_silverkite_helper import cols_interact
from greykite.common import constants as cst
from greykite.common.features.timeseries_features import build_time_features_df
from greykite.common.features.timeseries_features import convert_date_to_continuous_time
from greykite.framework.benchmark.data_loader_ts import DataLoaderTS
from greykite.framework.templates.autogen.forecast_config import EvaluationPeriodParam
from greykite.framework.templates.autogen.forecast_config import ForecastConfig
from greykite.framework.templates.autogen.forecast_config import MetadataParam
from greykite.framework.templates.autogen.forecast_config import ModelComponentsParam
from greykite.framework.templates.forecaster import Forecaster
from greykite.framework.templates.model_templates import ModelTemplateEnum
from greykite.framework.utils.result_summary import summarize_grid_search_results
# Loads dataset into UnivariateTimeSeries
dl = DataLoaderTS()
ts = dl.load_peyton_manning_ts()
df = ts.df # cleaned pandas.DataFrame
|
Exploratory data analysis (EDA)¶
After reading in a time series, we could first do some exploratory data analysis.
The UnivariateTimeSeries class is
used to store a timeseries and perform EDA.
122 123 124 | # describe
print(ts.describe_time_col())
print(ts.describe_value_col())
|
Out:
{'data_points': 2964, 'mean_increment_secs': 86400.0, 'min_timestamp': Timestamp('2007-12-10 00:00:00'), 'max_timestamp': Timestamp('2016-01-20 00:00:00')}
count 2905.000000
mean 8.138958
std 0.845957
min 5.262690
25% 7.514800
50% 7.997999
75% 8.580168
max 12.846747
Name: y, dtype: float64
The df has two columns, time column “ts” and value column “y”. The data is daily that ranges from 2007-12-10 to 2016-01-20. The data value ranges from 5.26 to 12.84
Let’s plot the original timeseries.
(The interactive plot is generated by plotly: click to zoom!)
134 135 | fig = ts.plot()
plotly.io.show(fig)
|
A few exploratory plots can be plotted to reveal the time series’s properties.
The UnivariateTimeSeries class
has a very powerful plotting tool
plot_quantiles_and_overlays.
A tutorial of using the function can be found at Seasonality.
Baseline model¶
A simple forecast can be created on the data set, see details in Simple Forecast. Note that if you do not provide any extra parameters, all model parameters are by default. The default parameters are chosen conservatively, so consider this a baseline model to assess forecast difficulty and make further improvements if necessary.
153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 | # Specifies dataset information
metadata = MetadataParam(
time_col="ts", # name of the time column
value_col="y", # name of the value column
freq="D" # "H" for hourly, "D" for daily, "W" for weekly, etc.
)
forecaster = Forecaster()
result = forecaster.run_forecast_config(
df=df,
config=ForecastConfig(
model_template=ModelTemplateEnum.SILVERKITE.name,
forecast_horizon=365, # forecasts 365 steps ahead
coverage=0.95, # 95% prediction intervals
metadata_param=metadata
)
)
|
Out:
Fitting 3 folds for each of 1 candidates, totalling 3 fits
For a detailed documentation about the output from
run_forecast_config,
see Check Forecast Result. Here we could plot the forecast.
176 177 178 | forecast = result.forecast
fig = forecast.plot()
plotly.io.show(fig)
|
Model performance evaluation¶
We can see the forecast fits the existing data well; however, we do not have a good ground truth to assess how well it predicts into the future.
Train-test-split¶
The typical way to evaluate model performance is to reserve part of the training data and use it to measure the model performance. Because we always predict the future in a time series forecasting problem, we reserve data from the end of training set to measure the performance of our forecasts. This is called a time series train test split.
By default, the results returned by run_forecast_config
creates a time series train test split and stores the test result in result.backtest.
The reserved testing data by default has the
same length as the forecast horizon. We can access the evaluation results:
199 | pd.DataFrame(result.backtest.test_evaluation, index=["Value"]).transpose() # formats dictionary as a pd.DataFrame
|
Evaluation metrics¶
From here we can see a list of metrics that measure the model performance on the test data. You may choose one or a few metrics to focus on. Typical metrics include:
- MSE:
Mean squared error, the average squared error. Could be affected by extreme values.
- RMSE:
Root mean squared error, the square root of MSE.
- MAE:
Mean absolute error, the average of absolute error. Could be affected by extreme values.
- MedAE:
Median absolute error, the median of absolute error. Less affected by extreme values.
- MAPE:
Mean absolute percent error, measures the error percent with respective to the true values. This is useful when you would like to consider the relative error instead of the absolute error. For example, an error of 1 is considered as 10% for a true observation of 10, but as 1% for a true observation of 100. This is the default metric we like.
- MedAPE:
Median absolute percent error, the median version of MAPE, less affected by extreme values.
Let’s use MAPE as our metric in this example. Looking at these results, you may have a basic sense of how the model is performing on the unseen test data. On average, the baseline model’s prediction is 11.3% away from the true values.
Time series cross-validation¶
Forecast quality depends a lot of the evaluation time window. The evaluation window selected above might happen to be a relatively easy/hard period to predict. Thus, it is more robust to evaluate over a longer time window when dataset size allows. Let’s consider a more general way of evaluating a forecast model: time series cross-validation.
Time series cross-validation is based on a time series rolling split. Let’s say we would like to perform an evaluation with a 3-fold cross-validation, The whole training data is split in 3 different ways. Since our forecast horizon is 365 days, we do:
- First fold:
Train from 2007-12-10 to 2013-01-20, forecast from 2013-01-21 to 2014-01-20, and compare the forecast with the actual.
- Second fold:
Train from 2007-12-10 to 2014-01-20, forecast from 2014-01-21 to 2015-01-20, and compare the forecast with the actual.
- Third fold:
Train from 2007-12-10 to 2015-01-20, forecast from 2015-01-21 to 2016-01-20, and compare the forecast with the actual.
The split could be more flexible, for example, the testing periods could have gaps. For more details about evaluation period configuration, see Evaluation Period. The forecast model’s performance will be the average of the three evaluations on the forecasts.
By default, the results returned by run_forecast_config
also runs time series cross-validation internally.
You are allowed to configure the cross-validation splits, as shown below.
Here note that the test_horizon are reserved from the back of
the data and not used for cross-validation.
This part of testing data can further evaluate the model performance
besides the cross-validation result, and is available for plotting.
268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 | # Defines the cross-validation config
evaluation_period = EvaluationPeriodParam(
test_horizon=365, # leaves 365 days as testing data
cv_horizon=365, # each cv test size is 365 days (same as forecast horizon)
cv_max_splits=3, # 3 folds cv
cv_min_train_periods=365 * 4 # uses at least 4 years for training because we have 8 years data
)
# Runs the forecast
result = forecaster.run_forecast_config(
df=df,
config=ForecastConfig(
model_template=ModelTemplateEnum.SILVERKITE.name,
forecast_horizon=365, # forecasts 365 steps ahead
coverage=0.95, # 95% prediction intervals
metadata_param=metadata,
evaluation_period_param=evaluation_period
)
)
# Summarizes the cv result
cv_results = summarize_grid_search_results(
grid_search=result.grid_search,
decimals=1,
# The below saves space in the printed output. Remove to show all available metrics and columns.
cv_report_metrics=None,
column_order=["rank", "mean_test", "split_test", "mean_train", "split_train", "mean_fit_time", "mean_score_time", "params"])
# Transposes to save space in the printed output
cv_results["params"] = cv_results["params"].astype(str)
cv_results.set_index("params", drop=True, inplace=True)
cv_results.transpose()
|
Out:
Fitting 3 folds for each of 1 candidates, totalling 3 fits
By default, all metrics in ElementwiseEvaluationMetricEnum
are computed on each CV train/test split.
The configuration of CV evaluation metrics can be found at
Evaluation Metric.
Here, we show the Mean Absolute Percentage Error (MAPE)
across splits (see summarize_grid_search_results
to control what to show and for details on the output columns).
From the result, we see that the cross-validation mean_test_MAPE is 7.3%, which
means the prediction is 7.3% away from the ground truth on average. We also see the
3 cv folds have split_test_MAPE 5.1%, 8.5% and 8.4%, respectively.
When we have different sets of model parameters, a good way to compare them is to run a time series cross-validation on each set of parameters, and pick the set of parameters that has the best cross-validated performance.
Start tuning¶
Now that you know how to evaluate model performance, let’s see if we can improve the model by tuning its parameters.
Anomaly¶
An anomaly is a deviation in the metric that is not expected to occur again in the future. Including anomaly points will lead the model to fit the anomaly as an intrinsic property of the time series, resulting in inaccurate forecasts. These anomalies could be identified through overlay plots, see Seasonality.
329 330 331 332 333 334 335 336 337 338 339 340 341 | fig = ts.plot_quantiles_and_overlays(
groupby_time_feature="month_dom",
show_mean=True,
show_quantiles=False,
show_overlays=True,
overlay_label_time_feature="year",
overlay_style={"line": {"width": 1}, "opacity": 0.5},
center_values=True,
xlabel="day of year",
ylabel=ts.original_value_col,
title="yearly seasonality for each year (centered)",
)
plotly.io.show(fig)
|
From the yearly overlay plot above, we could see two big anomalies:
one in March of 2012, and one in June of 2010. Other small anomalies
could be identified as well, however, they have less influence.
The SILVERKITE template currently supports masking anomaly points
by supplying the anomaly_info as a dictionary. You could
either assign adjusted values to them, or simply mask them as NA
(in which case these dates will not be used in fitting).
For a detailed introduction about the anomaly_info configuration,
see Examine Input Data.
Here we define an anomaly_df dataframe to mask them as NA,
and wrap it into the anomaly_info dictionary.
356 357 358 359 360 361 362 363 364 365 366 367 368 369 | anomaly_df = pd.DataFrame({
# start and end date are inclusive
# each row is an anomaly interval
cst.START_DATE_COL: ["2010-06-05", "2012-03-01"], # inclusive
cst.END_DATE_COL: ["2010-06-20", "2012-03-20"], # inclusive
cst.ADJUSTMENT_DELTA_COL: [np.nan, np.nan], # mask as NA
})
# Creates anomaly_info dictionary.
# This will be fed into the template.
anomaly_info = {
"value_col": "y",
"anomaly_df": anomaly_df,
"adjustment_delta_col": cst.ADJUSTMENT_DELTA_COL,
}
|
Adding relevant features¶
Growth and trend¶
First we look at the growth and trend. Detailed growth configuration can be found
at Growth.
In these two features, we care less about the short-term fluctuations but rather long-term tendency.
From the original plot we see there is no obvious growth pattern, thus we
could use a linear growth to fit the model. On the other hand, there could be
potential trend changepoints, at which time the linear growth changes its rate.
Detailed changepoint configuration can be found at Changepoints.
These points can be detected with the ChangepointDetector class. For a quickstart example,
see Changepoint detection.
Here we explore the automatic changepoint detection.
The parameters in this automatic changepoint detection is customized for this data set.
We keep the yearly_seasonality_order the same as the model’s yearly seasonality order.
The regularization_strength controls how many changepoints are detected.
0.5 is a good choice, while you may try other numbers such as 0.4 or 0.6 to see the difference.
The resample_freq is set to 7 days, because we have a long training history, thus we should
keep this relatively long (the intuition is that shorter changes will be ignored).
We put 25 potential changepoints to be the candidates, because we do not expect too many changes.
However, this could be higher.
The yearly_seasonality_change_freq is set to 365 days, which means we refit the yearly seasonality
every year, because it can be see from the time series plot that the yearly seasonality varies every year.
The no_changepoint_distance_from_end is set to 365 days, which means we do not allow any changepoints
at the last 365 days of training data. This avoids fitting the final trend with too little data.
For long-term forecast, this is typically the same as the forecast horizon, while for short-term forecast,
this could be a multiple of the forecast horizon.
402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 | model = ChangepointDetector()
res = model.find_trend_changepoints(
df=df, # data df
time_col="ts", # time column name
value_col="y", # value column name
yearly_seasonality_order=10, # yearly seasonality order, fit along with trend
regularization_strength=0.5, # between 0.0 and 1.0, greater values imply fewer changepoints, and 1.0 implies no changepoints
resample_freq="7D", # data aggregation frequency, eliminate small fluctuation/seasonality
potential_changepoint_n=25, # the number of potential changepoints
yearly_seasonality_change_freq="365D", # varying yearly seasonality for every year
no_changepoint_distance_from_end="365D") # the proportion of data from end where changepoints are not allowed
fig = model.plot(
observation=True,
trend_estimate=False,
trend_change=True,
yearly_seasonality_estimate=False,
adaptive_lasso_estimate=True,
plot=False)
plotly.io.show(fig)
|
From the plot we see the automatically detected trend changepoints. The results shows that the time series is generally increasing until 2012, then generally decreasing. One possible explanation is that 2011 is the last year Peyton Manning was at the Indianapolis Colts before joining the Denver Broncos. If we feed the trend changepoint detection parameter to the template, these trend changepoint features will be automatically included in the model.
430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 | # The following specifies the growth and trend changepoint configurations.
growth = {
"growth_term": "linear"
}
changepoints = {
"changepoints_dict": dict(
method="auto",
yearly_seasonality_order=10,
regularization_strength=0.5,
resample_freq="7D",
potential_changepoint_n=25,
yearly_seasonality_change_freq="365D",
no_changepoint_distance_from_end="365D"
)
}
|
Seasonality¶
The next features we will look into are the seasonality features. Detailed seasonality configurations can be found at Seasonality. A detailed seasonality detection quickstart example on the same data set is available at Seasonality Detection. The conclusions about seasonality terms are:
daily seasonality is not available (because frequency is daily);
weekly and yearly patterns are evident (weekly will also interact with football season);
monthly or quarterly seasonality is not evident.
Therefore, for pure seasonality terms, we include weekly and yearly seasonality. The seasonality orders are something to be tuned; here let’s take weekly seasonality order to be 5 and yearly seasonality order to be 10. For tuning info, see Seasonality.
465 466 467 468 469 470 471 472 473 474 475 | # Includes yearly seasonality with order 10 and weekly seasonality with order 5.
# Set the other seasonality to False to disable them.
yearly_seasonality_order = 10
weekly_seasonality_order = 5
seasonality = {
"yearly_seasonality": yearly_seasonality_order,
"quarterly_seasonality": False,
"monthly_seasonality": False,
"weekly_seasonality": weekly_seasonality_order,
"daily_seasonality": False
}
|
We will add the interaction between weekly seasonality and the football season
later in this tutorial.
The SILVERKITE template also supports seasonality changepoints. A seasonality
changepoint is a time point after which the periodic effect behaves
differently. For SILVERKITE, this means the Fourier series coefficients are allowed
to change. We could decide to add this feature if cross-validation performance is poor
and seasonality changepoints are detected in exploratory analysis.
For details, see Changepoint Detection.
Holidays and events¶
Then let’s look at holidays and events. Detailed holiday and event configurations can be found at Holidays and Events. Ask yourself which holidays are likely to affect the time series’ values. We expect that major United States holidays may affect wikipedia pageviews, since most football fans are in the United States. Events such as superbowl could potentially increase the pageviews. Therefore, we add United States holidays and superbowls dates as custom events. Other important events that affect the time series can also be found through the yearly seasonality plots in Seasonality.
500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 | # Includes major holidays and the superbowl date.
events = {
# These holidays as well as their pre/post dates are modeled as individual events.
"holidays_to_model_separately": SilverkiteHoliday.ALL_HOLIDAYS_IN_COUNTRIES, # all holidays in "holiday_lookup_countries"
"holiday_lookup_countries": ["UnitedStates"], # only look up holidays in the United States
"holiday_pre_num_days": 2, # also mark the 2 days before a holiday as holiday
"holiday_post_num_days": 2, # also mark the 2 days after a holiday as holiday
"daily_event_df_dict": {
"superbowl": pd.DataFrame({
"date": ["2008-02-03", "2009-02-01", "2010-02-07", "2011-02-06",
"2012-02-05", "2013-02-03", "2014-02-02", "2015-02-01", "2016-02-07"], # dates must cover training and forecast period.
"event_name": ["event"] * 9 # labels
})
},
}
|
Autoregression¶
The autoregressive features are very useful in short-term forecasting, but could be risky to use in long-term forecasting. Detailed autoregression configurations can be found at Auto-regression.
Custom¶
Now we consider some custom features that could relate to the pageviews. The documentation for
extra regressors can be found at Regressors. As mentioned
in Seasonality, we observe that the football
season heavily affects the pageviews, therefore we need to use regressors to identify the football season.
There are multiple ways to include this feature: adding indicator for the whole season;
adding number of days till season start (end) and number of days since season start (end).
The former puts a uniform effect over all in-season dates, while the latter quantify
the on-ramp and down-ramp. If you are not sure which effect to include, it’s ok to include both
effects. SILVERKITE has the option to use Ridge regression as the fit algorithm to avoid
over-fitting too many features. Note that many datetime features could also be added to
the model as features. SILVERKITE calculates some of these features, which can be added to
extra_pred_cols as an arbitrary patsy expression.
For a full list of such features, see build_time_features_df.
If a feature is not automatically created by SILVERKITE, we need to create it
beforehand and append it to the data df.
Here we create the “is_football_season” feature.
Note that we also need to provide the customized column for the forecast horizon period as well.
The way we do it is to first create the df with timestamps covering the forecast horizon.
This can be done with the make_future_dataframe
function within the UnivariateTimeSeries class.
Then we create a new column of our customized regressor for this augmented df.
548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 | # Makes augmented df with forecast horizon 365 days
df_full = ts.make_future_dataframe(periods=365)
# Builds "df_features" that contains datetime information of the "df"
df_features = build_time_features_df(
dt=df_full["ts"],
conti_year_origin=convert_date_to_continuous_time(df_full["ts"][0])
)
# Roughly approximates the football season.
# "woy" is short for "week of year", created above.
# Football season is roughly the first 6 weeks and last 17 weeks in a year.
is_football_season = (df_features["woy"] <= 6) | (df_features["woy"] >= 36)
# Adds the new feature to the dataframe.
df_full["is_football_season"] = is_football_season.astype(int).tolist()
df_full.reset_index(drop=True, inplace=True)
# Configures regressor column.
regressors = {
"regressor_cols": ["is_football_season"]
}
|
Interactions¶
Finally, let’s consider what possible interactions are relevant to the forecast problem.
Generally speaking, if a feature behaves differently on different values of another feature,
these two features could have potential interaction effects.
As in Seasonality, the weekly seasonality
is different through football season and non-football season, therefore, the multiplicative
term is_football_season x weekly_seasonality is able to capture this pattern.
579 580 581 582 583 584 585 586 587 588 589 590 591 | fig = ts.plot_quantiles_and_overlays(
groupby_time_feature="str_dow",
show_mean=True,
show_quantiles=False,
show_overlays=True,
center_values=True,
overlay_label_time_feature="month", # splits overlays by month
overlay_style={"line": {"width": 1}, "opacity": 0.5},
xlabel="day of week",
ylabel=ts.original_value_col,
title="weekly seasonality by month",
)
plotly.io.show(fig)
|
Now let’s create the interaction terms: interaction between is_football_season and weekly seasonality.
The interaction terms between a feature and a seasonality feature
can be created with the cols_interact function.
598 599 600 601 602 603 604 605 | football_week = cols_interact(
static_col="is_football_season",
fs_name=SilverkiteSeasonalityEnum.WEEKLY_SEASONALITY.value.name,
fs_order=weekly_seasonality_order,
fs_seas_name=SilverkiteSeasonalityEnum.WEEKLY_SEASONALITY.value.seas_names
)
extra_pred_cols = football_week
|
Moreover, the multiplicative term month x weekly_seasonality and the dow_woy features also
account for the varying weekly seasonality through the year. One could added these features, too.
Here we just leave them out. You may use cols_interact again to create the month x weekly_seasonality
similar to is_football_season x weekly_seasonality. dow_woy is automatically calcuated by SILVERKITE,
you may simply append the name to extra_pred_cols to include it in the model.
Putting things together¶
Now let’s put everything together and produce a new forecast.
A detailed template documentation can be found at
Configure a Forecast.
We first configure the MetadataParam class.
The MetadataParam class includes basic proporties of the time series itself.
622 623 624 625 626 627 628 | metadata = MetadataParam(
time_col="ts", # column name of timestamps in the time series df
value_col="y", # column name of the time series values
freq="D", # data frequency, here we have daily data
anomaly_info=anomaly_info, # this is the anomaly information we defined above,
train_end_date=datetime.datetime(2016, 1, 20)
)
|
Next we define the ModelComponentsParam class based on the discussion on relevant features.
The ModelComponentsParam include properties related to the model itself.
634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 | model_components = ModelComponentsParam(
seasonality=seasonality,
growth=growth,
events=events,
changepoints=changepoints,
autoregression=None,
regressors=regressors, # is_football_season defined above
uncertainty={
"uncertainty_dict": "auto",
},
custom={
# What algorithm is used to learn the relationship between the time series and the features.
# Regularized fitting algorithms are recommended to mitigate high correlations and over-fitting.
# If you are not sure what algorithm to use, "ridge" is a good choice.
"fit_algorithm_dict": {
"fit_algorithm": "ridge",
},
"extra_pred_cols": extra_pred_cols # the interaction between is_football_season and weekly seasonality defined above
}
)
|
Now let’s run the model with the new configuration. The evaluation config is kept the same as the previous case; this is important for a fair comparison of parameter sets.
660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 | # Runs the forecast
result = forecaster.run_forecast_config(
df=df_full,
config=ForecastConfig(
model_template=ModelTemplateEnum.SILVERKITE.name,
forecast_horizon=365, # forecasts 365 steps ahead
coverage=0.95, # 95% prediction intervals
metadata_param=metadata,
model_components_param=model_components,
evaluation_period_param=evaluation_period
)
)
# Summarizes the cv result
cv_results = summarize_grid_search_results(
grid_search=result.grid_search,
decimals=1,
# The below saves space in the printed output. Remove to show all available metrics and columns.
cv_report_metrics=None,
column_order=["rank", "mean_test", "split_test", "mean_train", "split_train", "mean_fit_time", "mean_score_time", "params"])
# Transposes to save space in the printed output
cv_results["params"] = cv_results["params"].astype(str)
cv_results.set_index("params", drop=True, inplace=True)
cv_results.transpose()
|
Out:
Fitting 3 folds for each of 1 candidates, totalling 3 fits
Now we see that after analyzing the problem and adding appropriate features, the cross-validation test MAPE is 5.4%, which is improved compared with the baseline (7.3%). The 3 cv folds also have their MAPE reduced to 3.9%, 8.7% and 3.8%, respectively. The first and third fold improved significantly. With some investigation, we can see that the second fold did not improve because there is a trend changepoint right at the the start of its test period.
It would be hard to know this situation until we see it. In the cross-validation step, one
way to avoid this is to set a different evaluation period. However, leaving this period
also makes sense because it could happen again in the future.
In the forecast period, we could monitor the forecast and actual, and re-train the model
to adapt to the most recent pattern if we see a deviation. In the changepoints dictionary,
tune regularization_strength or no_changepoint_distance_from_end accordingly, or
add manually specified changepoints to the automatically detected ones. For details, see
Changepoints.
We could also plot the forecast.
704 705 706 | forecast = result.forecast
fig = forecast.plot()
plotly.io.show(fig)
|
Check model summary¶
To further investigate the model mechanism, it’s also helpful
to see the model summary.
The ModelSummary module
provides model results such as estimations, significance, p-values,
confidence intervals, etc.
that can help the user understand how the model works and
what can be further improved.
The model summary is a class method of the estimator and can be used as follows.
721 722 | summary = result.model[-1].summary() # -1 retrieves the estimator from the pipeline
print(summary)
|
Out:
================================ Model Summary =================================
Number of observations: 2964, Number of features: 287
Method: Ridge regression
Number of nonzero features: 287
Regularization parameter: 2.848
Residuals:
Min 1Q Median 3Q Max
-2.092 -0.2289 -0.05127 0.1587 3.205
Pred_col Estimate Std. Err Pr(>)_boot sig. code 95%CI
Intercept 7.803 0.05644 <2e-16 *** (7.686, 7.902)
events_Christmas Day -0.3566 0.1254 0.002 ** (-0.5884, -0.08584)
events_C...bserved) -0.3852 0.3111 0.208 (-0.8557, 0.105)
events_C...erved)-1 -0.289 0.2895 0.354 (-0.7923, 0.2204)
events_C...erved)-2 -0.2097 0.255 0.496 (-0.6668, 0.2319)
events_C...erved)+1 0.008593 0.0656 0.796 (-0.1238, 0.1448)
events_C...erved)+2 0.04168 0.05834 0.450 (-0.06915, 0.1467)
events_C...as Day-1 -0.1885 0.1289 0.138 (-0.4367, 0.06679)
events_C...as Day-2 -0.07981 0.213 0.714 (-0.5551, 0.2314)
events_C...as Day+1 -0.2303 0.1297 0.060 . (-0.4505, 0.02074)
events_C...as Day+2 0.07615 0.08536 0.362 (-0.08939, 0.2498)
events_Columbus Day -0.1138 0.1435 0.420 (-0.3664, 0.184)
events_C...us Day-1 0.07423 0.07429 0.306 (-0.0837, 0.2019)
events_C...us Day-2 -0.05108 0.05669 0.368 (-0.1542, 0.07176)
events_C...us Day+1 0.01457 0.1025 0.882 (-0.1983, 0.2038)
events_C...us Day+2 0.005195 0.09423 0.960 (-0.172, 0.1981)
events_I...ence Day -0.03354 0.07261 0.652 (-0.1696, 0.114)
events_I...bserved) -0.1073 0.05373 0.032 * (-0.1951, 0.003502)
events_I...erved)-1 -0.1054 0.06844 0.104 (-0.2122, 0.02439)
events_I...erved)-2 -0.0639 0.05472 0.240 (-0.1565, 0.04552)
events_I...erved)+1 0.01105 0.06234 0.838 (-0.1003, 0.136)
events_I...erved)+2 0.08295 0.1152 0.550 (-0.08537, 0.303)
events_I...ce Day-1 -0.117 0.05382 0.032 * (-0.2072, -0.008786)
events_I...ce Day-2 -0.1045 0.05478 0.052 . (-0.2112, -0.001569)
events_I...ce Day+1 -0.03637 0.0668 0.614 (-0.1624, 0.0946)
events_I...ce Day+2 -0.03666 0.07672 0.644 (-0.1643, 0.129)
events_Labor Day -0.9081 0.1478 <2e-16 *** (-1.131, -0.5625)
events_Labor Day-1 -0.08142 0.1196 0.514 (-0.3226, 0.1372)
events_Labor Day-2 -0.07605 0.06104 0.212 (-0.1846, 0.04457)
events_Labor Day+1 -0.4698 0.0985 <2e-16 *** (-0.6568, -0.2792)
events_Labor Day+2 -0.1605 0.08005 0.046 * (-0.3088, -0.002947)
events_M... Jr. Day 0.2639 0.1948 0.180 (-0.1094, 0.6375)
events_M...r. Day-1 0.2341 0.2388 0.346 (-0.2359, 0.6765)
events_M...r. Day-2 -0.06938 0.1 0.502 (-0.2537, 0.1286)
events_M...r. Day+1 -0.09268 0.1438 0.538 (-0.3705, 0.1848)
events_M...r. Day+2 0.03705 0.114 0.726 (-0.1725, 0.252)
events_Memorial Day -0.172 0.0477 <2e-16 *** (-0.2528, -0.07478)
events_M...al Day-1 -0.1218 0.06764 0.064 . (-0.2245, 0.02881)
events_M...al Day-2 -0.06832 0.0861 0.422 (-0.2001, 0.1278)
events_M...al Day+1 -0.03727 0.05513 0.480 (-0.1451, 0.07694)
events_M...al Day+2 0.08794 0.08527 0.306 (-0.07131, 0.252)
events_New Years Day -0.1494 0.09138 0.116 (-0.3289, 0.04632)
events_N...bserved) 0.0206 0.04066 0.508 (-0.06218, 0.1023)
events_N...erved)-1 -0.0941 0.06173 0.098 . (-0.1994, 0.02044)
events_N...erved)-2 -0.04235 0.1313 0.676 (-0.2778, 0.2065)
events_N...erved)+1 0.1807 0.1189 0.086 . (-0.03229, 0.3591)
events_N...erved)+2 0.04302 0.04312 0.250 (-0.02837, 0.1414)
events_N...rs Day-1 -0.02544 0.08793 0.796 (-0.1901, 0.1377)
events_N...rs Day-2 0.1148 0.1121 0.320 (-0.1046, 0.3088)
events_N...rs Day+1 0.1684 0.07858 0.032 * (0.001665, 0.3156)
events_N...rs Day+2 0.1893 0.1298 0.146 (-0.07317, 0.4519)
events_Thanksgiving -0.07547 0.06912 0.282 (-0.1969, 0.05584)
events_T...giving-1 -0.2546 0.06322 <2e-16 *** (-0.3703, -0.1204)
events_T...giving-2 -0.2523 0.07681 <2e-16 *** (-0.3897, -0.09363)
events_T...giving+1 -0.04444 0.08339 0.572 (-0.2165, 0.118)
events_T...giving+2 -0.1189 0.05559 0.044 * (-0.2373, -0.008828)
events_Veterans Day -0.03341 0.05603 0.544 (-0.1487, 0.07381)
events_V...bserved) -0.1164 0.07937 0.066 . (-0.2343, 0.)
events_V...erved)-1 0.03432 0.03028 0.168 (-0.005292, 0.09466)
events_V...erved)-2 -0.008749 0.02174 0.430 (-0.05996, 0.03168)
events_V...erved)+1 -0.05363 0.04076 0.102 (-0.1295, 0.)
events_V...erved)+2 -0.04841 0.03595 0.096 . (-0.1175, 0.)
events_V...ns Day-1 -0.07678 0.06042 0.206 (-0.1842, 0.05673)
events_V...ns Day-2 -0.0182 0.07354 0.810 (-0.1467, 0.1353)
events_V...ns Day+1 -0.02022 0.06173 0.748 (-0.1483, 0.1053)
events_V...ns Day+2 -0.04613 0.04347 0.288 (-0.1255, 0.05135)
events_W...Birthday -0.0329 0.08266 0.684 (-0.1797, 0.1347)
events_W...rthday-1 -0.2622 0.07204 <2e-16 *** (-0.4023, -0.1192)
events_W...rthday-2 -0.1086 0.05426 0.056 . (-0.2107, 0.01002)
events_W...rthday+1 -0.05971 0.0599 0.292 (-0.1768, 0.06567)
events_W...rthday+2 -0.1126 0.03663 0.006 ** (-0.1792, -0.03931)
events_superbowl 0.4419 0.2532 0.096 . (-0.07024, 0.9164)
str_dow_2-Tue 0.02405 0.01413 0.078 . (-0.001907, 0.05353)
str_dow_3-Wed 0.01784 0.01176 0.126 (-0.005002, 0.0423)
str_dow_4-Thu 0.006795 0.01369 0.606 (-0.01864, 0.03547)
str_dow_5-Fri -0.02559 0.01209 0.026 * (-0.0487, -0.002014)
str_dow_6-Sat -0.0495 0.0102 <2e-16 *** (-0.06965, -0.02936)
str_dow_7-Sun -0.008702 0.01573 0.544 (-0.04157, 0.02162)
is_footb...w_weekly -0.05625 0.02212 0.010 * (-0.1059, -0.018)
is_footb...w_weekly 0.4787 0.02631 <2e-16 *** (0.4299, 0.5323)
is_footb...w_weekly -0.02021 0.01158 0.086 . (-0.04295, 0.0031)
is_footb...w_weekly 0.1007 0.01209 <2e-16 *** (0.07805, 0.1251)
is_footb...w_weekly -0.02886 0.01047 0.010 * (-0.0491, -0.008811)
is_footb...w_weekly 0.008454 0.01356 0.516 (-0.01557, 0.0362)
is_footb...w_weekly 0.02886 0.01047 0.010 * (0.008811, 0.0491)
is_footb...w_weekly 0.008454 0.01356 0.516 (-0.01557, 0.0362)
is_footb...w_weekly 0.02021 0.01158 0.086 . (-0.0031, 0.04295)
is_footb...w_weekly 0.1007 0.01209 <2e-16 *** (0.07805, 0.1251)
is_football_season 0.5158 0.0915 <2e-16 *** (0.3584, 0.7018)
ct1 -0.4201 0.06794 <2e-16 *** (-0.5566, -0.2816)
is_weekend:ct1 -0.06344 0.03776 0.096 . (-0.1376, 0.01361)
str_dow_2-Tue:ct1 -0.0147 0.06642 0.830 (-0.1523, 0.1081)
str_dow_3-Wed:ct1 -0.08275 0.04782 0.078 . (-0.1776, 0.004278)
str_dow_4-Thu:ct1 -0.03337 0.03951 0.402 (-0.1195, 0.03649)
str_dow_5-Fri:ct1 -0.01293 0.03624 0.716 (-0.08559, 0.05145)
str_dow_6-Sat:ct1 -0.02598 0.03999 0.500 (-0.1018, 0.04894)
str_dow_7-Sun:ct1 -0.03745 0.06591 0.556 (-0.1625, 0.09431)
cp0_2008_07_21_00 -0.2149 0.06271 0.002 ** (-0.3226, -0.07633)
is_weeke...07_21_00 -0.07226 0.03602 0.050 . (-0.1389, 0.001611)
str_dow_...07_21_00 -0.05792 0.03326 0.088 . (-0.1215, 0.0124)
str_dow_...07_21_00 -0.004965 0.02441 0.844 (-0.05095, 0.04476)
str_dow_...07_21_00 -0.0411 0.02461 0.088 . (-0.08443, 0.007375)
str_dow_...07_21_00 -0.03848 0.02552 0.146 (-0.08576, 0.01044)
str_dow_...07_21_00 -0.06369 0.02745 0.020 * (-0.1219, -0.01275)
str_dow_...07_21_00 -0.008568 0.03649 0.820 (-0.08058, 0.0652)
cp1_2008_11_10_00 1.003 0.06415 <2e-16 *** (0.8594, 1.111)
is_weeke...11_10_00 0.2044 0.03382 <2e-16 *** (0.132, 0.2639)
str_dow_...11_10_00 0.1291 0.04404 0.004 ** (0.04249, 0.204)
str_dow_...11_10_00 0.1554 0.03318 <2e-16 *** (0.08721, 0.218)
str_dow_...11_10_00 0.1136 0.03029 <2e-16 *** (0.05179, 0.1685)
str_dow_...11_10_00 0.1202 0.03677 <2e-16 *** (0.04825, 0.1893)
str_dow_...11_10_00 0.1074 0.03239 <2e-16 *** (0.04403, 0.1689)
str_dow_...11_10_00 0.09702 0.04373 0.026 * (0.00232, 0.173)
cp2_2009_03_09_00 0.3711 0.05939 <2e-16 *** (0.2617, 0.4877)
is_weeke...03_09_00 0.1121 0.0386 <2e-16 *** (0.03801, 0.1904)
str_dow_...03_09_00 0.0434 0.04895 0.402 (-0.05185, 0.128)
str_dow_...03_09_00 0.0452 0.03878 0.224 (-0.03012, 0.1211)
str_dow_...03_09_00 0.0313 0.03407 0.362 (-0.03048, 0.105)
str_dow_...03_09_00 0.04474 0.03953 0.260 (-0.03128, 0.1195)
str_dow_...03_09_00 0.07753 0.04009 0.070 . (-0.001445, 0.1623)
str_dow_...03_09_00 0.0346 0.05363 0.518 (-0.07366, 0.1358)
cp3_2009_10_19_00 -0.4704 0.07417 <2e-16 *** (-0.6063, -0.3233)
is_weeke...10_19_00 -0.1084 0.04369 0.012 * (-0.1914, -0.03089)
str_dow_...10_19_00 -0.0759 0.05719 0.194 (-0.1894, 0.03387)
str_dow_...10_19_00 -0.0874 0.04456 0.044 * (-0.174, -0.005855)
str_dow_...10_19_00 -0.04999 0.03716 0.168 (-0.116, 0.02139)
str_dow_...10_19_00 -0.05887 0.04483 0.192 (-0.1497, 0.02409)
str_dow_...10_19_00 -0.04535 0.04168 0.262 (-0.1247, 0.0377)
str_dow_...10_19_00 -0.06306 0.04753 0.192 (-0.154, 0.02708)
cp4_2010_02_15_00 -0.477 0.08177 <2e-16 *** (-0.6281, -0.3217)
is_weeke...02_15_00 -0.14 0.05298 0.006 ** (-0.2454, -0.03967)
str_dow_...02_15_00 -0.05004 0.05163 0.342 (-0.1567, 0.0467)
str_dow_...02_15_00 -0.08166 0.04548 0.072 . (-0.1766, 0.0005712)
str_dow_...02_15_00 -0.04409 0.03931 0.272 (-0.1214, 0.02844)
str_dow_...02_15_00 -0.08251 0.04949 0.094 . (-0.1886, 0.006913)
str_dow_...02_15_00 -0.07317 0.04661 0.122 (-0.1632, 0.01427)
str_dow_...02_15_00 -0.06685 0.05544 0.254 (-0.1714, 0.0441)
cp5_2010_06_07_00 0.1524 0.0708 0.034 * (-0.0005831, 0.2806)
is_weeke...06_07_00 0.04952 0.04327 0.252 (-0.0322, 0.1325)
str_dow_...06_07_00 0.005808 0.04374 0.908 (-0.08132, 0.09025)
str_dow_...06_07_00 0.05367 0.03641 0.138 (-0.02342, 0.1185)
str_dow_...06_07_00 0.04172 0.03406 0.220 (-0.02996, 0.1129)
str_dow_...06_07_00 -0.001987 0.04294 0.964 (-0.08569, 0.07184)
str_dow_...06_07_00 0.0292 0.03955 0.460 (-0.05145, 0.1022)
str_dow_...06_07_00 0.02032 0.04751 0.656 (-0.06763, 0.1182)
cp6_2011_01_24_00 0.2651 0.08179 <2e-16 *** (0.1293, 0.4357)
is_weeke...01_24_00 0.06135 0.0566 0.264 (-0.04631, 0.1726)
str_dow_...01_24_00 0.02369 0.06733 0.716 (-0.1173, 0.152)
str_dow_...01_24_00 0.0421 0.0474 0.376 (-0.04331, 0.1338)
str_dow_...01_24_00 0.01953 0.0398 0.624 (-0.05166, 0.1059)
str_dow_...01_24_00 0.03062 0.05468 0.582 (-0.07544, 0.1366)
str_dow_...01_24_00 0.02935 0.04765 0.538 (-0.05755, 0.1204)
str_dow_...01_24_00 0.032 0.06896 0.640 (-0.1187, 0.1559)
cp7_2011_05_16_00 0.3743 0.07819 <2e-16 *** (0.2299, 0.5299)
is_weeke...05_16_00 0.06395 0.04962 0.208 (-0.02468, 0.1678)
str_dow_...05_16_00 0.07994 0.0421 0.054 . (0.003412, 0.1668)
str_dow_...05_16_00 0.0699 0.0437 0.104 (-0.02653, 0.1565)
str_dow_...05_16_00 0.03932 0.04099 0.340 (-0.04554, 0.1176)
str_dow_...05_16_00 0.05649 0.05569 0.314 (-0.0518, 0.1606)
str_dow_...05_16_00 0.01349 0.05367 0.820 (-0.09515, 0.1189)
str_dow_...05_16_00 0.05045 0.05377 0.346 (-0.05963, 0.1444)
cp8_2012_01_02_00 -0.2182 0.09592 0.020 * (-0.4337, -0.05906)
is_weeke...01_02_00 0.009077 0.05062 0.872 (-0.1027, 0.09805)
str_dow_...01_02_00 -0.05135 0.06572 0.420 (-0.1917, 0.07088)
str_dow_...01_02_00 -0.07913 0.05221 0.140 (-0.1776, 0.02412)
str_dow_...01_02_00 -0.08078 0.0532 0.120 (-0.1934, 0.01048)
str_dow_...01_02_00 0.0531 0.07677 0.510 (-0.1088, 0.1827)
str_dow_...01_02_00 -0.009237 0.05606 0.858 (-0.1245, 0.08599)
str_dow_...01_02_00 0.01831 0.06128 0.800 (-0.1047, 0.1357)
cp9_2012_04_23_00 -0.9381 0.08727 <2e-16 *** (-1.1, -0.744)
is_weeke...04_23_00 -0.1823 0.05324 <2e-16 *** (-0.2797, -0.07082)
str_dow_...04_23_00 -0.1463 0.05774 0.016 * (-0.2698, -0.04929)
str_dow_...04_23_00 -0.1643 0.055 0.004 ** (-0.275, -0.05528)
str_dow_...04_23_00 -0.1162 0.04922 0.018 * (-0.2227, -0.0323)
str_dow_...04_23_00 -0.1539 0.05124 <2e-16 *** (-0.2442, -0.04102)
str_dow_...04_23_00 -0.1028 0.05349 0.062 . (-0.2119, 0.002198)
str_dow_...04_23_00 -0.07953 0.05657 0.156 (-0.1943, 0.02861)
cp10_2012_08_13_00 0.06176 0.08697 0.462 (-0.1166, 0.2199)
is_weeke...08_13_00 -0.07427 0.04228 0.076 . (-0.1517, 0.01056)
str_dow_...08_13_00 0.01243 0.05935 0.816 (-0.1201, 0.12)
str_dow_...08_13_00 0.0877 0.03878 0.014 * (0.005398, 0.1546)
str_dow_...08_13_00 0.06894 0.05009 0.188 (-0.02947, 0.1559)
str_dow_...08_13_00 -0.04406 0.04918 0.352 (-0.1369, 0.06086)
str_dow_...08_13_00 0.0215 0.03847 0.566 (-0.05302, 0.0944)
str_dow_...08_13_00 -0.09577 0.04902 0.046 * (-0.1886, -0.003089)
cp11_2013_04_01_00 0.6097 0.06021 <2e-16 *** (0.4923, 0.7151)
is_weeke...04_01_00 0.1213 0.04176 0.002 ** (0.03845, 0.2018)
str_dow_...04_01_00 0.113 0.05478 0.046 * (0.01385, 0.2326)
str_dow_...04_01_00 0.07494 0.05037 0.128 (-0.0243, 0.1693)
str_dow_...04_01_00 0.07154 0.04987 0.156 (-0.02143, 0.1769)
str_dow_...04_01_00 0.09219 0.05306 0.082 . (-0.0223, 0.1868)
str_dow_...04_01_00 0.05036 0.05138 0.334 (-0.04886, 0.1456)
str_dow_...04_01_00 0.07096 0.06775 0.282 (-0.07567, 0.1985)
cp12_2014_03_10_00 -0.4588 0.05766 <2e-16 *** (-0.5667, -0.3437)
is_weeke...03_10_00 -0.07332 0.03672 0.046 * (-0.1473, -0.007792)
str_dow_...03_10_00 -0.01682 0.08019 0.848 (-0.1786, 0.1481)
str_dow_...03_10_00 -0.09497 0.05511 0.090 . (-0.2077, 0.00971)
str_dow_...03_10_00 -0.05342 0.04882 0.300 (-0.1431, 0.04586)
str_dow_...03_10_00 -0.03822 0.0555 0.454 (-0.1487, 0.07291)
str_dow_...03_10_00 -0.07655 0.06031 0.210 (-0.1905, 0.0419)
str_dow_...03_10_00 0.003228 0.07545 0.964 (-0.1453, 0.1366)
ct1:sin1_tow_weekly -0.04642 0.04582 0.316 (-0.1454, 0.03792)
ct1:cos1_tow_weekly -0.1795 0.0823 0.034 * (-0.3496, -0.02396)
ct1:sin2_tow_weekly 0.06279 0.05671 0.266 (-0.05421, 0.1696)
ct1:cos2_tow_weekly -0.1322 0.0757 0.082 . (-0.2815, 0.009589)
cp0_2008...w_weekly 0.01753 0.04596 0.676 (-0.07344, 0.1078)
cp0_2008...w_weekly 0.0454 0.06013 0.446 (-0.07926, 0.164)
cp0_2008...w_weekly -0.07155 0.0512 0.160 (-0.1763, 0.029)
cp0_2008...w_weekly 0.02691 0.0552 0.630 (-0.07848, 0.1365)
cp1_2008...w_weekly 0.06903 0.04675 0.138 (-0.02185, 0.1535)
cp1_2008...w_weekly 0.1525 0.06828 0.028 * (0.00677, 0.2859)
cp1_2008...w_weekly 0.01559 0.0504 0.764 (-0.07096, 0.1202)
cp1_2008...w_weekly 0.1392 0.06516 0.022 * (0.01049, 0.2559)
cp2_2009...w_weekly -0.03046 0.05177 0.554 (-0.1324, 0.07024)
cp2_2009...w_weekly 0.04712 0.0655 0.458 (-0.07856, 0.175)
cp2_2009...w_weekly 0.03312 0.06186 0.608 (-0.08247, 0.1537)
cp2_2009...w_weekly 0.01379 0.06265 0.810 (-0.111, 0.1393)
cp3_2009...w_weekly -0.04718 0.0657 0.474 (-0.166, 0.09137)
cp3_2009...w_weekly -0.04884 0.08328 0.568 (-0.2087, 0.129)
cp3_2009...w_weekly -0.001207 0.06636 0.988 (-0.13, 0.1252)
cp3_2009...w_weekly -0.007164 0.08046 0.908 (-0.1603, 0.15)
cp4_2010...w_weekly 0.02153 0.07116 0.772 (-0.1116, 0.1668)
cp4_2010...w_weekly -0.003041 0.08264 0.982 (-0.1644, 0.1596)
cp4_2010...w_weekly -0.009966 0.07253 0.892 (-0.1619, 0.1216)
cp4_2010...w_weekly 0.007884 0.07968 0.926 (-0.1542, 0.1688)
cp5_2010...w_weekly 0.03148 0.05801 0.596 (-0.08587, 0.1435)
cp5_2010...w_weekly -0.03428 0.06489 0.620 (-0.1563, 0.09543)
cp5_2010...w_weekly -0.05894 0.06078 0.332 (-0.1629, 0.06343)
cp5_2010...w_weekly -0.05203 0.05943 0.394 (-0.1601, 0.05987)
cp6_2011...w_weekly 0.001124 0.0734 0.986 (-0.1387, 0.1377)
cp6_2011...w_weekly 0.06146 0.08802 0.476 (-0.1252, 0.223)
cp6_2011...w_weekly -0.004963 0.0822 0.972 (-0.1585, 0.1537)
cp6_2011...w_weekly 0.04231 0.07795 0.574 (-0.1156, 0.1931)
cp7_2011...w_weekly 0.0706 0.06795 0.304 (-0.06015, 0.192)
cp7_2011...w_weekly 0.04115 0.07237 0.582 (-0.07914, 0.1876)
cp7_2011...w_weekly 0.0177 0.07355 0.826 (-0.1248, 0.1621)
cp7_2011...w_weekly 0.02031 0.06913 0.752 (-0.1072, 0.1617)
cp8_2012...w_weekly -0.1807 0.07578 0.020 * (-0.3207, -0.0211)
cp8_2012...w_weekly -0.04509 0.1057 0.660 (-0.2693, 0.1649)
cp8_2012...w_weekly 0.06708 0.08294 0.434 (-0.09999, 0.2329)
cp8_2012...w_weekly 0.000601 0.09953 0.988 (-0.1831, 0.1849)
cp9_2012...w_weekly -0.09577 0.07353 0.202 (-0.2358, 0.04587)
cp9_2012...w_weekly -0.01311 0.08255 0.888 (-0.1836, 0.1444)
cp9_2012...w_weekly -0.06793 0.07921 0.380 (-0.22, 0.08243)
cp9_2012...w_weekly -0.05262 0.08092 0.488 (-0.22, 0.09944)
cp10_201...w_weekly 0.1982 0.06704 0.002 ** (0.04704, 0.3102)
cp10_201...w_weekly -0.08764 0.09162 0.336 (-0.2798, 0.08764)
cp10_201...w_weekly -0.01159 0.0808 0.896 (-0.1783, 0.1414)
cp10_201...w_weekly -0.0533 0.08355 0.516 (-0.2262, 0.08817)
cp11_201...w_weekly 0.04788 0.0562 0.410 (-0.04475, 0.1683)
cp11_201...w_weekly 0.07602 0.07354 0.322 (-0.05979, 0.2173)
cp11_201...w_weekly 0.04648 0.06614 0.484 (-0.07624, 0.177)
cp11_201...w_weekly 0.08498 0.06688 0.204 (-0.04678, 0.2139)
cp12_201...w_weekly -0.04022 0.04694 0.378 (-0.1351, 0.04428)
cp12_201...w_weekly -0.06984 0.08182 0.384 (-0.2317, 0.09197)
cp12_201...w_weekly 0.0003325 0.05793 0.992 (-0.1052, 0.1092)
cp12_201...w_weekly -0.08167 0.07443 0.240 (-0.2312, 0.06547)
sin1_tow_weekly 0.1053 0.03371 <2e-16 *** (0.04108, 0.1745)
cos1_tow_weekly 0.06865 0.04881 0.166 (-0.02602, 0.1618)
sin2_tow_weekly -0.0226 0.02265 0.324 (-0.06535, 0.02311)
cos2_tow_weekly 0.04849 0.02452 0.044 * (0.0004629, 0.09497)
sin3_tow_weekly -0.00687 0.01324 0.568 (-0.03035, 0.02176)
cos3_tow_weekly 0.005713 0.02081 0.780 (-0.03507, 0.04979)
sin4_tow_weekly 0.00687 0.01324 0.568 (-0.02176, 0.03035)
cos4_tow_weekly 0.005713 0.02081 0.780 (-0.03507, 0.04979)
sin5_tow_weekly 0.0226 0.02265 0.324 (-0.02311, 0.06535)
cos5_tow_weekly 0.04849 0.02452 0.044 * (0.0004629, 0.09497)
sin1_ct1_yearly -0.04426 0.02228 0.052 . (-0.08619, 0.00257)
cos1_ct1_yearly 0.4169 0.05425 <2e-16 *** (0.31, 0.5157)
sin2_ct1_yearly 0.1042 0.01417 <2e-16 *** (0.07665, 0.134)
cos2_ct1_yearly -0.1489 0.01557 <2e-16 *** (-0.1811, -0.1217)
sin3_ct1_yearly 0.1752 0.01612 <2e-16 *** (0.1434, 0.205)
cos3_ct1_yearly -0.01013 0.01352 0.468 (-0.03405, 0.01744)
sin4_ct1_yearly -0.04372 0.01581 0.002 ** (-0.07649, -0.0146)
cos4_ct1_yearly -0.0897 0.013 <2e-16 *** (-0.1133, -0.0637)
sin5_ct1_yearly -0.06317 0.01288 <2e-16 *** (-0.08745, -0.03771)
cos5_ct1_yearly -0.01998 0.01077 0.068 . (-0.04047, 0.001532)
sin6_ct1_yearly -0.06777 0.01454 <2e-16 *** (-0.0934, -0.03856)
cos6_ct1_yearly -0.009463 0.01222 0.432 (-0.03254, 0.01441)
sin7_ct1_yearly -0.04924 0.011 <2e-16 *** (-0.0701, -0.02769)
cos7_ct1_yearly 0.02889 0.0114 0.010 * (0.006343, 0.04969)
sin8_ct1_yearly 0.01905 0.01276 0.136 (-0.004705, 0.04409)
cos8_ct1_yearly 0.06879 0.01272 <2e-16 *** (0.04526, 0.0933)
sin9_ct1_yearly -0.004435 0.01188 0.686 (-0.0278, 0.01866)
cos9_ct1_yearly -0.03218 0.01173 0.004 ** (-0.05248, -0.00851)
sin10_ct1_yearly -0.06601 0.01168 <2e-16 *** (-0.09072, -0.04185)
cos10_ct1_yearly -0.0234 0.01134 0.040 * (-0.04601, -0.00107)
Signif. Code: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Multiple R-squared: 0.7553, Adjusted R-squared: 0.744
F-statistic: 64.947 on 130 and 2832 DF, p-value: 1.110e-16
Model AIC: 18720.0, model BIC: 19511.0
WARNING: the condition number is large, 1.42e+05. This might indicate that there are strong multicollinearity or other numerical problems.
WARNING: the F-ratio and its p-value on regularized methods might be misleading, they are provided only for reference purposes.
The model summary shows the model information, the coefficients and their significance, and a few summary statistics. For example, we can see the changepoints and how much the growth rate changes at each changepoint. We can see that some of the holidays have significant effect in the model, such as Christmas, Labor day, Thanksgiving, etc. We can see the significance of the interaction between football season and weekly seasonality etc.
For a more detailed guide on model summary, see Model Summary.
Summary in model tuning¶
After the example, you may have some sense about how to select parameters and tune the model. Here we list a few steps and tricks that might help select the best models. What you may do:
Detect anomaly points with the overlay plots (
plot_quantiles_and_overlays). Mask these points with NA. Do not specify the adjustment unless you are confident about how to correct the anomalies.Choose an appropriate way to model the growth (linear, quadratic, square root, etc.) If none of the typical growth shape fits the time series, you might consider linear growth with trend changepoints. Try different changepoint detection configurations. You may also plot the detected changepoints and see if it makes sense to you. The template also supports custom changepoints. If the automatic changepoint detection result does not make sense to you, you might supply your own changepoints.
Choose the appropriate seasonality orders. The higher the order, the more details the model can learn. However, too large orders could overfit the training data. These can also be detected from the overlay plots (
plot_quantiles_and_overlays). There isn’t a unified way to choose seasonality, so explore different seasonality orders and compare the results.Consider what events and holidays to model. Are there any custom events we need to add? If you add a custom event, remember also adding the dates for the event in the forecast period.
Add external regressors that could be related to the time series. Note that you will need to provide the values of the regressors in the forecast period as well. You may use another time series as a regressor, as long as you have a ground truth/good forecast for it that covers your forecast period.
Adding interaction terms. Let’s mention again here that there could be interaction between two features if the behaviors of one feature are different when the other feature have different values. Try to detect this through the overlay plot (
plot_quantiles_and_overlays), too. By default, we have a few pre-defined interaction terms, see feature_set_enabled.Choose an appropriate fit algorithm. This is the algorithm that models the relationship between the features and the time series. See a full list of available algorithms at fit_algorithm. If you are unsure about their difference, try some of them and compare the results. If you don’t want to, choosing “ridge” is a safe option.
It is worth noting that the template supports automatic grid search with different sets of parameters. For each parameter, if you provide the configuration in a list, it will automatically run each combination and choose the one with the best cross-validation performance. This will save a lot of time. For details, see grid search.
Follow your insights and intuitions, and play with the parameters, you will get good forecasts!
Total running time of the script: ( 3 minutes 50.472 seconds)