Change point detection in time series data plays a crucial role across various domains. In a previous blog post, we showcased the application of Bayesian Change Point Detection using the Python machine learning client for SAP HANA(hana-ml). However, this method primarily focuses on analyzing complete time series data, and its robust decomposition capabilities can demand significant computational resources. To address the increasing need for continuous monitoring and efficient change point identification in time series data, we are thrilled to introduce our novel approach: Online Bayesian Change Point Detection with Memory-Saving Mechanism.

In this blog post, you will learn:

• How does the memory-saving mechanism work.
• How to apply online change point detection in hana-ml to extract the change point continuously.

Introduction

Contrary to the offline version, online Bayesian change point detection does not rely on decomposition techniques. It emphasizes causal predictive filtering, a method that generates an accurate distribution of the upcoming, unseen data point in the sequence using only the information from previously observed data.

In conventional methods for continuous change point detection in the incoming time series, we rely on models that incorporate the entire historical dataset. However, this approach leads to a significant issue: as the volume of data increases, the model size grows excessively large, requiring substantial time to read and output the model repeatedly in a tabular format. Consequently, this method becomes inefficient for online scenarios.

Based on our experiments, we have determined that an efficient strategy is to remove all historical data before the most recent change point. In the upcoming section, we will provide step-by-step instructions on implementing this approach with two datasets—one originating from a simulated scenario and the other from a real-world example.

Examples

All source code in examples of the following context will use Python machine learning client for SAP HANA Predictive Analsysi Library(PAL).

Connect to SAP HANA

import hana_ml
from hana_ml import dataframe
cc = dataframe.ConnectionContext(address='', port=30x15, user='', password='')#account details omitted

Use Case I:  Mocking Data

In this use case, we will focus on continuously detecting the change points in the mocking data.

The simulated dataset consists of two segments: the first segment has a length of 40, and the second spans 60 units. These segments are separated by a red vertical line in the graph. The data is stored in two separate database tables named ‘PAL_ONLINE_BCPD_MOCK_TBL_PART1’ and ‘PAL_ONLINE_BCPD_MOCK_TBL_PART2’. To create corresponding hana_ml.DataFrame objects for these tables, we can utilize the table() function of the ConnectionContext.

mocking_df_part1 = cc.table('PAL_ONLINE_BCPD_MOCK_TBL_PART1')
mocking_df_part2 = cc.table('PAL_ONLINE_BCPD_MOCK_TBL_PART2')

from hana_ml.algorithms.pal.tsa.changepoint import OnlineBCPD
obcpd_with_prune    = OnlineBCPD(threshold=0.5, prune=True)
obcpd_without_prune = OnlineBCPD(threshold=0.5, prune=False)

model_with_prune, cp_part1_with_prune = obcpd_with_prune.fit_predict(data=conn.table("PAL_ONLINE_BCPD_MOCK_TBL_PART1"), model=None)
model_without_prune, cp_part1_without_prune = obcpd_without_prune.fit_predict(data=conn.table("PAL_ONLINE_BCPD_MOCK_TBL_PART1"), model=None)

And we will get two output tables from each OnlineBCPD object, let’s print the length of each model:

print("The length of the model with prune is {}\nand the legnth of model without prune is {}".format(len(model_with_prune.collect()), len(model_without_prune.collect())))

The collect() function of hana_ml.DataFrame can help to fetch data from database and you will see the output like this:

By setting ‘prune=True’, the model will consume less memory while executing the algorithm.

Afterwards, we print the change point table:

print(cp_part1_with_prune.collect())
print(cp_part1_without_prune.collect())

and we will see the output like this:

which shows that the results are the same under both settings regarding parameter ‘prune’.

Further, we are to use the pruned model to detect the second part:

model_with_prune, cp_part2 = obcpd_with_prune.fit_predict(data=conn.table("PAL_ONLINE_BCPD_MOCK_TBL_PART2"), model=model_with_prune)
print(cp_part2.collect())

The results should be like this:

Finally, we plot all the change points in the original graph by orange vertical lines:

It’s done! All the change points in two continuous segements have be detected.

Use Case II: Real World Sensor Data

In this example, we have a real-world sensor dataset with a length of 1500 and it is stored in a dataframe named df_sensor:

We divide it equally into 15 segments, each containing 100 data points. Then, we use the OnlineBCPD algorithm to continuously detect change points within the dataset using the following code snippet:

def createTableFromDataFrame(df, table_name, cc):    
    dropTable(table_name, cc)
    dt_ml = dataframe.create_dataframe_from_pandas(cc, df, table_name = table_name)
    return dt_ml

obcpd = OnlineBCPD(threshold=0.5, prune=True)
figure(num=None, figsize=(16, 4), dpi=80, facecolor='w', edgecolor='k')
plt.plot(data)
for i in range(15):
    createTableFromDataFrame(df_sensor[i * 100 : (i + 1) * 100], "PAL_ONLINE_BCPD_SENSOR_DATA_TBL", conn)
    model_with_prune, cp = obcpd_with_prune.fit_predict(data=conn.table("PAL_ONLINE_BCPD_SENSOR_DATA_TBL"))
    for pos in cp.collect()["POSITION"]:
        plt.axvline(x=pos, color='orange', linestyle='dashed')

Finally, we will get the following graph:

Discussion and Summary