Simple Anomaly Detection

You can create and evaluate an anomaly detection model with just a few lines of code.

Provide your timeseries as a pandas dataframe with timestamp and value. Optionally, you can also provide the anomaly labels as a column in the dataframe.

For example, to detect anomalies in daily sessions data, your dataframe could look like this:

import pandas as pd
df = pd.DataFrame({
    "date": ["2020-01-08-00", "2020-01-09-00", "2020-01-10-00"],
    "sessions": [10231.0, 12309.0, 12104.0],
    "is_anomaly": [False, True, False]
})

The time column can be any format recognized by pandas.to_datetime.

In this example, we’ll load a dataset representing log(daily page views) on the Wikipedia page for Peyton Manning. It contains values from 2007-12-10 to 2016-01-20. More dataset info here.

import warnings

import plotly
from greykite.common.data_loader import DataLoader
from greykite.detection.detector.config import ADConfig
from greykite.detection.detector.data import DetectorData
from greykite.detection.detector.greykite import GreykiteDetector
from greykite.framework.templates.autogen.forecast_config import ForecastConfig
from greykite.framework.templates.autogen.forecast_config import MetadataParam
from greykite.framework.templates.model_templates import ModelTemplateEnum

warnings.filterwarnings("ignore")

# Loads dataset into pandas DataFrame
dl = DataLoader()
df = dl.load_peyton_manning()

# specify dataset information
metadata = MetadataParam(
    time_col="ts",  # name of the time column ("date" in example above)
    value_col="y",  # name of the value column ("sessions" in example above)
    freq="D"  # "H" for hourly, "D" for daily, "W" for weekly, etc.
    # Any format accepted by `pandas.date_range`
)

Create an Anomaly Detection Model

Similar to forecasting, you need to provide a forecast config and an anomaly detection config. You can choose any of the available forecast model templates (see Choose a Model).

# In this example, we choose the "AUTO" model template for the forecast config,
# and the default anomaly detection config.
# The Silverkite "AUTO" model template chooses the parameter configuration
# given the input data frequency, forecast horizon and evaluation configs.

anomaly_detector = GreykiteDetector()  # Creates an instance of the Greykite anomaly detector

forecast_config = ForecastConfig(
    model_template=ModelTemplateEnum.AUTO.name,
    forecast_horizon=7,  # forecasts 7 steps ahead
    coverage=None,       # Confidence Interval will be tuned by the AD model
    metadata_param=metadata)

ad_config = ADConfig()  # Default anomaly detection config

detector = GreykiteDetector(
    forecast_config=forecast_config,
    ad_config=ad_config,
    reward=None)

Train the Anomaly Detection Model

You can train the anomaly detection model by calling the fit method. This method takes a DetectorData object as input. The DetectorData object consists the time series information as a pandas dataframe. Optionally, you can also provide the anomaly labels as a column in the dataframe. The anomaly labels can also be provided as a list of boolean values. The anomaly labels are used to evaluate the model performance.

train_size = int(2700)
df_train = df[:train_size].reset_index(drop=True)
train_data = DetectorData(df=df_train)
detector.fit(data=train_data)

Out:

Fitting 3 folds for each of 1 candidates, totalling 3 fits

Predict with the Anomaly Detection Model

You can predict anomalies by calling the predict method.

test_data = DetectorData(df=df)
test_data = detector.predict(test_data)

Evaluate the Anomaly Detection Model

The output of the anomaly detection model are stored as attributes of the GreykiteDetector object. (The interactive plots are generated by plotly: click to zoom!)

Training

The fitted_df attribute contains the result on the training data. You can plot the result by calling the plot method with phase="train".

print(detector.fitted_df)

fig = detector.plot(
    phase="train",
    title="Greykite Detector Peyton Manning - fit phase")
plotly.io.show(fig)

Out:

             ts    actual  forecast  forecast_lower  forecast_upper  is_anomaly_predicted   z_score is_anomaly
0    2007-12-10  9.590761  9.194596        9.166271        9.222921                  True  0.877043       None
1    2007-12-11  8.519590  9.022015        8.993690        9.050340                  True -1.112284       None
2    2007-12-12  8.183677  8.945369        8.917044        8.973694                  True -1.686259       None
3    2007-12-13  8.072467  8.895533        8.867208        8.923858                  True -1.822130       None
4    2007-12-14  7.893572  8.836495        8.808170        8.864820                  True -2.087475       None
...         ...       ...       ...             ...             ...                   ...       ...        ...
2753 2015-06-24  7.344073  6.915129        6.886804        6.943454                  True  0.949609       None
2754 2015-06-25  7.291656  6.937793        6.909468        6.966118                  True  0.783395       None
2755 2015-06-26  7.271704  6.872038        6.843713        6.900363                  True  0.884793       None
2756 2015-06-27  7.454720  6.485441        6.457116        6.513766                  True  2.145821       None
2757 2015-06-28  6.692084  6.709412        6.681087        6.737737                 False -0.038362       None

[2758 rows x 8 columns]

Prediction

The pred_df attribute contains the predicted result. You can plot the result by calling the plot method with phase="predict".

print(detector.pred_df)

fig = detector.plot(
    phase="predict",
    title="Greykite Detector Peyton Manning - predict phase")
plotly.io.show(fig)