Learning from Labeled Anomalies for Efficient Anomaly Detection using Python Machine Learning Client for SAP HANA

In a few separate blog posts, we have discussed the problem of anomaly detection in dataset with multiple features using techniques like one-class classification, clustering(DBSCAN) as well as statistical tests. However, all the aforementioned techniques become less applicable when the dataset of interest is of high dimensionality(i.e. contains many features), or the boundary between normal points and anomalous ones is complicated. In this case, a better approach is to manually label the point of anomalies in the dataset, and then train a supervised machine learning model for the classification of normal points and anomalies.

In this blog post, we will analyze a specific dataset with labeled anomalies, and use the decision tree algorithm in SAP HANA Prective Analysis Library(PAL) through Python Machine Learning Client for SAP HANA(hana_ml) to construct a classification model for anomaly detection. In the meantime, several resampling techniques are also involved for improving the performance of the trained model on different perspectives.

Introduction

Separating anomalies from normal ones using with labeled datasets seems as simple as a regular classification problem. However, the highly skewed distribution between the normal and anomalous data points can pose a big challenge for building up any efficient classification model, because in datasets anomalies are usually so rare to be observed while normality is overwhelming. For example, we consider a disease with prevalence rate 0.1%, if we use a naive model that predict all people as non-patient of this disease, then this model has a “high” accuracy rate of 99.9%, seemingly good. However, if we happily adopt this naive model for detecting this disease, then it would be a disaster for all real patients. The imbalance of distribution between normal and anomalous labels is one typical characteristic for anomaly detection problems, especially when normal points and anomalies are entangled in the feature space of the dataset.

In the meatime, anomalous cases are usually much more valuable than normal ones. For example, if we fail to detect a fraudulent transaction between bank acounts, then we could have a great loss of money; however, if we suspect one transaction is fraudulent and it turn out to be not true, we only pay for some manual verification procedures that are realtively cheap. Higher importance for anomalous cases compared to normal ones is another typical characteristic for anomaly detection.

In this blog post, we will do a case study on anomaly detection using labeled dataset, the following contents will be included in our discussion:

1. Introduction & background knowledge on the dataset for our case study, with brief problem analysis
2. Anomaly detection from classification models with the help of various resampling techniques

Case Study: Thyroid Hyperfunctionality Detection by Classification

Dataset Description and Problem Analysis

The problem of interest in this blog post is thyroid disease recognition. The original full dataset is available in the UCI machine learning repository[1]. The original dataset contains 21 attributes – 15 of them are categorical and 6 are numerical. The dataset is divided into 3 classes : normal, subnormal and hyperfunction. Hyperfunction is the minority class in this dataset, but it is also the case that we are mostly interested in because once gained, it may accelerates the body’s metabolism, bringing along symptoms like unintentional loss of weight, rapid or irregular heartbeat, nervousness, anxiety and irritability, etc.

Our designated task in this blog post is to distinguish hyperfunctional cases from non-hyperfunctional(i.e. normal and subnormal) ones, using only the 6 numerical attributes.  A reduced version of this dataset for this target is available in the ODDS library[2], where all attribute values are scaled into the range [0, 1] using Min-Max scalar. Besides, The label column in this dataset is valued with 0s and 1s, where 1 for hyperfunction and 0 for non-hyperfunction.

Let us examine the corresponding dataset for further analysis.  We assume that the data has already been stored in a table with name ‘PAL_THYROID_DATA_TBL’ in a database of SAP HANA platform. Then the dataset can be accessed by establishing a connection to the database using hana_ml, illustrated as follows:

import hana_ml
from hana_ml.dataframe import ConnectionContext
cc = ConnectionContext('xx.xxx.xxx.xx', 30x15, 'XXX', 'Xxxxxxx')
thyroid_df = cc.table('PAL_THYROID_DATA_TBL')

Then, thyroid_df  a hana_ml.DataFrame that contains the information of the dataset, a brief description of this dataset could be obtained as follows:

thyroid_df.describe().collect()

As revealed by the mean value of the TYPE column, hyperfunctional cases covers less than 3% of all cases in the dataset, so the dataset is highly skewed w.r.t. thyroid functionality types.

Now we inspect the hyperfunctional cases and non-hyperfunctional cases separately.

cc.sql('select * from ({}) where TYPE=1'.format(thyroid_df.select_statement)).describe().collect()

cc.sql('select * from ({}) where TYPE=0'.format(thyroid_df.select_statement)).describe().collect()

A closer examination of the data tells us that the distribution of hyperfunctional cases and non-hyperfunctional ones are quite different(in quantiles) in attributes like V1, V2, V3, and V5. So these numerical attributes have the potential to tell the differences between thyroid hyperfunctionality and non-hyperfunctionality.

Visulization of the dataset is another way to assess the how difficult the classification problem should be. Dimensionality reduction is indispensible for the dataset since the its feature space is of dimension 6. Here PCA is applied for tranforming the dataset from 6D to 2D for drawing a scatter plot of the dataset(without involving the TYPE column).

from hana_ml.algorithms.pal.decomposition import PCA
res = PCA().fit_transform(thyroid_df, key='ID',
                          features=['V0', 'V1',
                                    'V2', 'V3',
                                    'V4', 'V5'])

thyroid_2d = res[['ID', 'COMPONENT_1', 'COMPONENT_2']]
hyper_ids = 'SELECT ID FROM ({}) WHERE TYPE=1'.format(thyroid_df.select_statement)
non_hyper_ids = 'SELECT ID FROM ({}) WHERE TYPE=0'.format(thyroid_df.select_statement)
thyroid_2d_hyper = cc.sql('SELECT * FROM ({}) WHERE ID IN ({})'.format(thyroid_2d.select_statement, hyper_ids))
thyroid_2d_non_hyper = cc.sql('SELECT * FROM ({}) WHERE ID IN ({})'.format(thyroid_2d.select_statement, non_hyper_ids))

import matplotlib.pyplot as plt
h1 = plt.scatter(thyroid_2d_non_hyper.collect()['COMPONENT_1'], thyroid_2d_non_hyper.collect()['COMPONENT_2'], c='red')
h2 = plt.scatter(thyroid_2d_hyper.collect()['COMPONENT_1'], thyroid_2d_hyper.collect()['COMPONENT_2'], s = 5, c='blue')
plt.legend((h1, h2), ('Normal/Subnormal', 'Hyperfunction'))
plt.show()

One can see from the above figure that hyperfunctional cases are distributed differently from the non-hyperfunctional ones(in the reduced attribute space), yet the two classes are not that well separated.

Dataset Partition

Before bulding up any classfication model for anomaly detection, we firstly divide the whole dataset into training and testing part using the train_test_val_split() method in hana_ml, where training percent is set to 0.7 and testing percent is set to 0.3(no validation data).

rom hana_ml.algorithms.pal.partition import train_test_val_split
train_, test_, _ = train_test_val_split(data = thyroid_df,
                                        id_column='ID',
                                        random_seed=2,
                                        partition_method='stratified',
                                        stratified_column='TYPE',
                                        training_percentage=0.7,
                                        testing_percentage=0.3,
                                        validation_percentage=0)

from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
dtc = UnifiedClassification(func='DecisionTree', algorithm='cart')
dtc.fit(data=train_,
        partition_method='no',
        key='ID',
        label='TYPE',
        categorical_variable='TYPE')
res_direct = dtc.score(data=test_,
                       key='ID',
                       features=['V0','V1', 'V2',
                                 'V3', 'V4', 'V5'])

Values of evaluation metrics of the trained model on the test dataset is available in the 2nd element of returned result of the score() function, and it can be collected to the python client as follows:

res_direct[1].collect()

A few key statistical values that worth mentioning:

1. The overall accuracy is ~99.38%, higher than the naive ‘always non-hyperfunctional’ classifier.
2. ~85.7% hyperfunctional cases in the test are assigned the correct label by the decision-tree classifier
3. ~88.9% predicted hyperfunctional cases of the decision-tree classifier are real hyperfunctional cases.

The performance of the trained model is already reasonably well. However, since classes are imbalanced in the training dataset, resampling techniques that balance the counts of classes have the potential for further improving performance of the classification model. We will verify this justification in the subsequent subsections. Our first try is to oversample the minority class for achieving class balance in the training data.

Model Training with Minority-class Oversampling

We firstly augment the number of hyperfunctional cases  several times in the training dataset using the synthetic minority over-sampling technique(i.e. SMOTE), so that the numer of hyperfunctional cases and that of non-hyperfunctional cases become comparable.

train_pos = cc.sql('SELECT * FROM ({}) WHERE TYPE=1'.format(train_.select_statement))
train_neg = cc.sql('SELECT * FROM ({}) WHERE TYPE=0'.format(train_.select_statement))
multi = int(train_neg.count()/train_pos.count())#times for count of non-hyperfunctional cases relative to that of hyperfunctional cases  

from hana_ml.algorithms.pal.preprocessing import SMOTE
smote = SMOTE(smote_amount=multi*100) #smote_amount is reflected by percentage 
train_smote = smote.fit_transform(data = train_[['V0', 'V1', 'V2', 'V3', 'V4', 'V5', 'TYPE']],
                                  label = 'TYPE',
                                  minority_class=1)

dtc.fit(data=train_smote,
        partition_method='no',
        label='TYPE',
        categorical_variable='TYPE')
score_res = dtc.score(data=test_,
                      key='ID',
                      features=['V0', 'V1', 'V2',
                                'V3', 'V4', 'V5'])

score_res[1].collect()

So, by oversampling the hyperfunctional cases in the traininig data to be roughly the same size as that of non-hyperfunctional cases, and retraining the decision tree model, we see the some improvement revealed by the following key statistics:

1. The overall accuracy increases from ~99.38% to ~99.73%(versus training without resampling)
2. ~96.4% hyperfunctional cases in the test are assigned the correct label by the decision-tree classifier(versus 85.7% without resampling)
3. ~93.1% predicted hyperfunctional cases of the decision-tree classifier are real hyperfunctional cases(versus 88.9% without resampling)

The increment of model performance on test dataset is non-negligible, which shows the effectivity of oversampling the minority-class.

Another approach for over-samping the minority class is direct duplication, which equivalent to bootstrapping is some sense.

import numpy as np
train_mult = train_neg
for i in range(np.int64(mult)):
    train_mult = train_mult.union(train_pos)#duplicate positive cases to several times so that their group are roughly the same size as negative group

dtc = UnifiedClassification(func='DecisionTree', algorithm='cart')
dtc.fit(data=train_mult[['V0', 'V1', 'V2', 'V3', 'V4', 'V5', 'TYPE']],
        partition_method='no',
        label='TYPE',
        categorical_variable='TYPE')
score_res = dtc.score(data=test_,
                      key='ID',
                      features=['V0', 'V1', 'V2', 'V3', 'V4', 'V5'])

score_res[1].collect()

from hana_ml.algorithms.pal.preprocessing import TomekLinks
tmk = TomekLinks()
train_tmk = tmk.fit_transform(train_, label='TYPE')

dtc.fit(data=train_tmk,
        key = 'ID',
        partition_method='no',
        label='TYPE',
        categorical_variable='TYPE')
score_res = dtc.score(data=test_,
                      key='ID',
                      features=['V0', 'V1', 'V2',
                                'V3', 'V4', 'V5'])

score_res[1].collect()

from hana_ml.algorithms.pal.preprocessing import Sampling
dsp_rate=0.03 # subsampling rate for the non-hyperfunctional cases
sp = Sampling(method='simple_random_without_replacement', sampling_size=int(dsp_rate*train_neg.count()), random_state=2)#
train_neg_sub = sp.fit_transform(train_neg)
train_balanced = train_neg_sub.union(train_pos)

dtc.fit(data=train_balanced,
        key='ID',
        partition_method='no',
        label='TYPE',
        categorical_variable='TYPE')
score_res = dtc.score(data=test_,
                      key='ID',
                      features=['V0', 'V1', 'V2',
                                'V3', 'V4', 'V5'])

score_res[1].collect()

from hana_ml.algorithms.pal.preprocessing import SMOTETomek
stk = SMOTETomek(smote_amount=multi*100)
train_stk = stk.fit_transform(data=train_,
                              label='TYPE',
                              minority_class=1)

from hana_ml.algorithms.pal.unified_classification import UnifiedClassification
dtc = UnifiedClassification(func='DecisionTree',
                            algorithm='cart')
dtc.fit(data=train_stk[['V0', 'V1', 'V2', 'V3', 'V4', 'V5', 'TYPE']],
        partition_method='no',
        label='TYPE',
        categorical_variable='TYPE')
score_res = dtc.score(data=test_,
                      key='ID',
                      features=['V0', 'V1', 'V2',
                                'V3', 'V4', 'V5'])

score_res[1].collect()

from hana_ml.algorithms.pal.preprocessing import Sampling
dsp_rate=0.5
sp = Sampling(method='simple_random_without_replacement',
              sampling_size=int(dsp_rate*train_neg.count()),
              random_state=2)
mult = int(train_neg.count()/train_pos.count())
train_neg_sub = sp.fit_transform(train_neg)
from hana_ml.algorithms.pal.preprocessing import SMOTE
smote = SMOTE(smote_amount=int(multi*100*dsp_rate))
train_pos_sup = smote.fit_transform(train_pos,
                                    label='TYPE',
                                    minority_class=1)
train_balanced = train_neg_sub.union(train_pos_sup)

dtc.fit(data=train_balanced,
        key='ID',
        partition_method='no',
        label='TYPE',
        categorical_variable='TYPE')
score_res = dtc.score(data=test_,
                      key='ID',
                      features=['V0', 'V1', 'V2',
                                'V3', 'V4', 'V5'])
score_res[1].collect()

References

[1] Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science
[2] Shebuti Rayana (2016). ODDS Library [http://odds.cs.stonybrook.edu]. Stony Brook, NY: Stony Brook University, Department of Computer Science
[3] Yang Feng, Min Zhou and Xin Tong (2020), Imbalanced classification: an objective-oriented review, Technical Report, New York University, School of Global Public Health