Outlier Detection with One-class Classification using Python Machine Learning Client for SAP HANA

We have introduced several methods for outlier detection in a few separate blog posts, inclusive of outlier detection using statistical tests and clustering. Typically, these methods can only detect outliers the input dataset, and the detection result cannot be generalized to new data points, because they do not come up with any model. Classification methods can be adopted to overcome this difficulty. However, the power of generalization for classification method does not come for free since they require the input data to be labeled.  As to the case of outlier detection, it means that any point in the input data must be labeled either as an inlier or an outlier.

Usually, classification for outlier detection requires the dataset to contain both inliers and outliers. However,  there are some cases where there could be no outliers in the input dataset, yet a model for outlier detection is still required. In such cases, one-class classification could be adopted for utlization. As its name suggests, one-class classification requires the input(i.e. training) data to be labeled by a single class, yet the trained model is also able to produce the label of opposing class like other binary classification models.

One-class support vector machine(i.e. one-class SVM) is perhaps the most frequently used method for one-class classification. This method is provided in SAP HANA Predictive Analysis Library(PAL) and wrapped up by the Python machine learning client for SAP HANA(hana_ml), and in this blog post it shall be adopted to solve the outlier detection problem.

Basically, after reading this blog post, you will learn:

• How to build a one-class SVM model given a dataset
• How to apply the trained model to the prediction data, extract the information of detected outliers and evaluate the performance of the trained model

Introduction

Suppose a new system has been set up, and it runs very smoothly in the early phase. In the meantime,  we want to build up a monitoring system to detect whether there are malfunctions as the new system keeps running. Then, what can we do? Waiting there until the system makes some mistake sounds a terrible idea, so we should be able to use the normal data in hand for initializing the monitoring system. There are plenty of similar cases in real life, where one-class classification becomes applicable.

Different from traditional classification methods, one-class classification tries to explore the inherent structure of the training dataset with a single label, and build a model for containing or characterizing the the training dataset, so that when a new point comes it can tell whether thr point is similar to points in the training dataset or is different from them.  One-class classification methods could be derived via either density estimation, or boundary estimation, or reconstruction mode estimation w.r.t. the input data. Among them, one-class SVM is a  boundary-estimation based method.

Basically, for outlier detection using one-class SVM, in the training phase a profile is drawn to encircle(almost) all points in the input data(all being inliers); while in the prediction phase, if a sample point falls into the region enclosed by the profile drawn it will be treated as an inlier, otherwise it will be treated an outlier.

In the following context of this blog post, we show a detailed case study of network intrusion detection using one-class SVM, where attacks are taking as outliers and normal connections as inliers.

The Case Study : One-class SVM for Network Intrusion Detection

import hana_ml
from hana_ml.dataframe import ConnectionContext
cc = ConnectionContext('xxx.xxx.xxx.xx', 30x15, 'XXX', 'Xxxxxxxx')# server info hidden away

http_data = cc.table('HTTP_DATA_TBL')
http_data.describe().collect()

normal_connections = cc.sql('SELECT * FROM ({}) WHERE Y = 0'.format(http_data.select_statement))
network_attacks = cc.sql('SELECT * FROM ({}) WHERE Y = 1'.format(http_data.select_statement))

normal_connections.describe().collect()

network_attacks.describe().collect()

from hana_ml.algorithms.pal.partition import train_test_val_split
train_normal, test_normal, _ = train_test_val_split(data=normal_connections,
                                                    id_column='ID',
                                                    random_seed=2,
                                                    training_percentage=0.9,
                                                    testing_percentage=0.1,
                                                    validation_percentage=0)
from hana_ml.algorithms.pal.svm import OneClassSVM
#nu is an upper bound on the fraction of training error,
#use a value close to the real(or empirically guessed) proportion of outliers 
osvm = OneClassSVM(kernel='rbf',
                   nu=0.003)
osvm.fit(data=train_normal,
         key='ID',
         features=['X0', 'X1', 'X2'])

normal_res = osvm.predict(data=test_normal,
                          key='ID',
                          features=['X0', 'X1', 'X2'])
attack_res = osvm.predict(data=network_attacks,
                          key='ID',
                          features=['X0', 'X1', 'X2'])

attack_res.collect()

normal_res.collect()

attack_correct = cc.sql('SELECT * FROM ({}) WHERE SCORE=-1'.format(attack_res.select_statement)).count()
attack_wrong = attack_data.count() - attack_correct

normal_correct = cc.sql('SELECT * FROM ({}) WHERE SCORE=1'.format(normal_res.select_statement)).count()
normal_wrong = normal_res.count() - normal_correct

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = attack_correct / (attack_correct + normal_wrong) = 0.8766851704996035

𝑅𝑒𝑐𝑎𝑙𝑙 = attack_correct / attack_data.count()= 1.0

𝐹1 = 2 × Precision × Recall / (Precision + Recall) = 0.9342911472638918

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = normal_correct / (normal_correct + attack_wrong) = 1.0

𝑅𝑒𝑐𝑎𝑙𝑙 = normal_correct / normal_data.count() = 0.9944983990518141

𝐹1 = 2 × Precision × Recall / (Precision + Recall) = 0.9972416117502018