Check Forecast Result
The output of run_forecast_config
helps you evaluate and interpret the forecast result.
from greykite.framework.templates.autogen.forecast_config import ForecastConfig
from greykite.framework.templates.autogen.forecast_config import MetadataParam
from greykite.framework.templates.forecaster import Forecaster
from greykite.framework.templates.model_templates import ModelTemplateEnum
metadata = MetadataParam(
time_col="ts",
value_col="y",
freq="D"
)
forecaster = Forecaster()
result = forecaster.run_forecast_config(
df=df, # input data
config=ForecastConfig(
model_template=ModelTemplateEnum.SILVERKITE.name,
metadata_param=metadata,
forecast_horizon=30,
coverage=0.95
)
)
# `result` is also stored as `forecaster.forecast_result`
The function returns a ForecastResult
object.
This is a dataclass with the forecast results. Below, we explain how to check each attribute
of the class.
Timeseries
timeseries
is your original input data, loaded into
a UnivariateTimeSeries
object.
The time series is represented in a standard format with convenient plotting functions and statistics.
If MetadataParam.anomaly_info
is specified, it contains the values before and after
anomaly adjustment.
For details, see Examine Input Data.
ts = result.timeseries # access the time series
Cross validation
grid_search
contains the cross validation results.
See Evaluation Period for cross validation configuration.
Use summarize_grid_search_results
to summarize the results. While grid_search.cv_results_
could be converted into
a pandas.DataFrame
without this function, the following conveniences
are provided:
returns the correct ranks based on each metric’s greater_is_better direction.
summarizes the hyperparameter space, only showing the parameters that change
combines split scores into a tuple to save table width
rounds the values to specified decimals
orders columns by type (test score, train score, metric, etc.)
from greykite.common.evaluation import EvaluationMetricEnum
from greykite.framework.utils.result_summary import summarize_grid_search_results
grid_search = result.grid_search
# default parameters
summarize_grid_search_results(
grid_search=grid_search,
only_changing_params=True,
combine_splits=True,
decimals=None,
score_func=EvaluationMetricEnum.MeanAbsolutePercentError.name,
score_func_greater_is_better=False,
cv_report_metrics=CV_REPORT_METRICS_ALL,
column_order=None
)
# some customization
metric = EvaluationMetricEnum[evaluation_metric.cv_selection_metric]
summarize_grid_search_results(
grid_search=grid_search,
decimals=2 # rounds the evaluation metrics
# List of regex to order the columns in the returned dataframe
column_order=["rank", "mean_test", "split_test", "mean_train", "split_train",
"mean_fit_time", "mean_score_time", "params", ".*"])
)
grid_search
is an instance of sklearn.model_selection.RandomizedSearchCV
, see
documentation for details on its attributes.
Warning
The ranks (e.g. “rank_test_MAPE”) in grid_search.cv_results_
should be ignored.
They are reversed for metrics where greater_is_better=False
, because sklearn
always assumes higher values are better when calculating the ranks, we report
metrics with their original sign.
Instead, use summarize_grid_search_results
to extract the CV results with proper ranks.
Backtest
backtest
checks the forecast accuracy on historical data. It is an instance
of UnivariateForecast
.
The forecast is computed on a holdout test set toward the end of the input dataset.
The length of the holdout test set is configured by test_horizon
in the template
configuration. See Evaluation Period.
You can plot the results:
backtest = result.backtest
fig = backtest.plot()
fig.show()
Show the evaluation metrics:
print(backtest.train_evaluation) # backtest training set
print(backtest.test_evaluation) # hold out test set
See the component plot to understand how factors like trend, seasonality, and holidays are handled by the forecast and to visualize changes points and residuals:
fig = backtest.plot_components()
fig.show()
Access backtest forecasted values and prediction intervals:
backtest.df
You can use backtest.plot_grouping_evaluation()
to examine the train/test error by
various dimensions (e.g. over time, by day of week). See
UnivariateForecast
for details.
See EvaluationMetricEnum
for the available error metrics
and their descriptions.
from greykite.common.evaluation import EvaluationMetricEnum
# MAPE by day of week
fig = backtest.plot_grouping_evaluation(
score_func=EvaluationMetricEnum.MeanAbsolutePercentError.get_metric_func(),
score_func_name=EvaluationMetricEnum.MeanAbsolutePercentError.get_metric_name(),
which="train", # "train" or "test" set
groupby_time_feature="dow", # day of week
groupby_sliding_window_size=None,
groupby_custom_column=None)
fig.show()
# RMSE over time
fig = backtest.plot_grouping_evaluation(
score_func=EvaluationMetricEnum.RootMeanSquaredError.get_metric_func(),
score_func_name=EvaluationMetricEnum.RootMeanSquaredError.get_metric_name(),
which="test", # "train" or "test" set
groupby_time_feature=None,
groupby_sliding_window_size=7, # weekly aggregation of daily data
groupby_custom_column=None)
fig.show()
See plot_flexible_grouping_evaluation
for a more powerful plotting function to plot the quantiles of the error along with the mean.
You can use component plots for a concise visual representation of how factors like the dataset’s trend, seasonality
and holiday patterns are estimated by the forecast model and to visualize other model information like residuals and
change points. Currently, by default, Silverkite
calculates component plots based on a dataset passed to the fit
method, whereas Prophet
calculates component plots based on a dataset passed to the predict
method. See the
discussion below on model forecasts for more information on how to access component plots based on the dataset
passed to the predict
method in Silverkite
.
backtest.plot_components()
Forecast
forecast
contains the forecasted values and the fit on the input dataset.
It is an instance of UnivariateForecast
.
After creating a forecast, plot it to see if it looks reasonable.
forecast = result.forecast
fig = forecast.plot()
fig.show()
Show the error metrics on the training set.
print(forecast.train_evaluation) # fit on the entire input dataset
Access future forecasted values and prediction intervals:
forecast.df
Just as for backtest, you can use forecast.plot_grouping_evaluation()
to examine the training error by
various dimensions (e.g. over time, by day of week), and forecast.plot_components()
to check the trend,
seasonality and holiday effects and visualize change points and residuals. Note that
components from the data set used in the predict
call can be accessed by .plot_components(predict_phase=True)
.
See UnivariateForecast
for details.
Model
model
is a sklearn.pipeline.Pipeline
object. It was
fit to the full dataset, with the best parameters selected via CV.
This model was used to generate forecast
. You can use it to extract
fitted model information (Model Summary)
or make another forecast (Simple Forecast).
model = result.model # access the model