Skip to Content
Technical Articles

Deep Learning using SAP Leonardo ML Foundation: Image Classification

This is the first article in a series of articles to come in the space of Deep Learning and how to use SAP Leonardo ML Foundation for the same. These articles will cover the complete process of a deep learning project starting for data preparation to prediction.

The first is a very popular and most successful application of deep learning: Image Classification.

Problem Statement: Make a deep learning model to classify images into various categories.

Dataset: Any image classification dataset. In this article, we will use the famous Fashion MNIST dataset. This dataset contains 70000 (60000 train + 10000 test) images of clothes having each image belonging to one of the 10 categories. You can get the dataset from here.  On unzipping, you will find train.zip and test.zip. The ten categories are:

  1. T-shirt/top.
  2. Trouser
  3. Pullover
  4. Dress
  5. Coat
  6. Sandal
  7. Shirt
  8. Sneaker
  9. Bag
  10. Ankle boot

Technological stack:

  1. Platform: Train Your Own Model functionality of SAP Leonardo ML Foundation.
  2. Deep Learning library: Keras.
  3. Programming language: Python 2

Let’s start the work! Major steps are:

  1. Data preparation.
  2. Uploading data to SAP Leonardo ML foundation.
  3. Making model and training.
  4. Making predictions using the trained model.

 

Data Preparation

The images are 28*28 in size, they are already very small, so we need not crop them. Also, images are black and white already.

But we should still do data augmentation as more the data, better the performance for deep learning.

The dataset structure is:

– fashion_mnist.zip

– train.zip

– train/

– train.csv

– test.zip

– test/

– test.csv

 

The train folder has 60000 images and test folder has 10000 images. We will first divide this dataset to train (85%=51000 images) and validation (15%=9000 images). Also, we will create a separate subfolder for each of the 10 categories in both train and validation folder. The directory structure should look like this:

-train

-0

-1

-9

-validation

-0

-1

-9

 

Let’s do this on our local systems first and then upload it to SAP Leonardo ML foundation. The python code to do this (run the code in the folder where data is unzipped):

import pandas as pd
import cv2
import os

train = pd.read_csv('train.csv')

os.mkdir('train_folders')
os.mkdir('validation_folders')

for i in range(10):
    images = train['id'][train['label'] == i].tolist()
    num_images = len(images)
    
    os.mkdir('train_folders/' + str(i))
    for image in images[:int(0.85*num_images)]:
        img = cv2.imread('train/' + str(image)+ '.png')
        cv2.imwrite('train_folders/' + str(i) + '/' + str(image) + '.png', img)
        
    os.mkdir('validation_folders/' + str(i))
    for image in images[int(0.85*num_images):]:
        img = cv2.imread('train/' + str(image)+ '.png')
        cv2.imwrite('validation_folders/' + str(i) + '/' + str(image) + '.png', img)

 

This code will result in two folders named train_folders and validation_folders. These folders should contain 10 folders each numbered from 0 to 9 having images of the folder name class.

Uploading data to SAP Leonardo ML foundation:

  1. Open command prompt and login to your respective space.
  2. Upload the data:
    cf sapml fs put train_folders fashion_mnist/​
    cf sapml fs put validation_folders fashion_mnist/

 

This will create a folder named fashion_mnist and two folders in it in SAP Leonardo ML foundation.

 

Making model and training:

Since the dataset is not huge, we will use pre-trained models from keras trained on the imagenet dataset. The idea is to retrain the pre-trained models like ResNet50 or InceptionV3 which are already trained on the imagenet dataset and fine tune them with our dataset. Here, I won’t go in details of fine tuning of models, but we will see how to accomplish this in code on SAP Leonardo ML foundation.

Let’s write code for training:

All the necessary libraries are imported. Note that we are going to use inception_resnet_v2 model from keras.

## necessary imports
import pandas as pd
import numpy as np

import keras
from keras.preprocessing.image import ImageDataGenerator
from keras.applications.inception_resnet_v2 import preprocess_input
from keras.models import Model, load_model
from keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau
from keras import optimizers, losses, activations, models
from keras.layers import Convolution2D, Dense, Input, Flatten, Dropout, MaxPooling2D, BatchNormalization, GlobalMaxPool2D

 

Let’s define some constants

## some constants
SIZE = 28
NUM_CLASSES = 10
BATCH_SIZE = 16
SEED = 1

train_data_dir = '../../fashion_mnist/train_folders/'
validation_data_dir = '../../fashion_mnist/validation_folders/'

nb_train_samples = 51000
nb_validation_samples = 9000

 

An image data generator is used to generate images from training folders, apply augmentation and feeds to the learning model. Here, we define two image generators, one for training and other for validation.

image_gen = ImageDataGenerator(rotation_range=15,
width_shift_range=0.2,
height_shift_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
rescale=1./255,
preprocessing_function=preprocess_input)

val_image_gen = ImageDataGenerator(rescale=1./255, preprocessing_function=preprocess_input)

train_gen = image_gen.flow_from_directory(train_data_dir, batch_size=BATCH_SIZE, seed=SEED, target_size=(SIZE, SIZE), shuffle=True, class_mode = "categorical")

valid_gen = val_image_gen.flow_from_directory(validation_data_dir, batch_size=BATCH_SIZE, seed=SEED, target_size=(SIZE, SIZE), shuffle=True, class_mode = "categorical")

 

Defining the model. Here, we will load the pre-trained inception_resnet_v2 add a couple of dense layers and dropout layers and compile with SGD optimizer. We will keep all the layers except added ones as non-trainable because they are already pre-trained. This function returns the compiled model.

def get_model():

    keras.backend.clear_session()

    model = keras.applications.inception_resnet_v2.InceptionResNetV2(include_top=False, 
    weights='imagenet', input_tensor=None, input_shape=(SIZE, SIZE, 3), pooling='avg')

    y = Dense(32, activation='relu')(model.layers[-1].output)
    y = Dropout(0.3)(y)
    y = Dense(32, activation='relu')(y)
    y = Dropout(0.2)(y)
    y = Dense(NUM_CLASSES, activation='softmax')(y)
    model = Model(inputs=model.input, outputs=y)

    for l in model.layers[:-1]:
        l.trainable = False

    sgd = keras.optimizers.SGD(lr=0.001, momentum=0.9)
    model.compile(optimizer=sgd, loss=losses.categorical_crossentropy, metrics=['acc'])
    model.summary()
    return model

 

Also, define some callbacks: checkpoint, early stopping and reducing learning rate.

filepath = '../../fashion_mnist/model-inception-resnet-v2-retrain-{epoch:02d}-{val_loss:.2f}-{val_acc:.4f}.hdf5'

checkpointer = ModelCheckpoint(filepath, verbose=1, save_best_only=True)
early = EarlyStopping(monitor='val_acc', min_delta=0, patience=3, verbose=1, mode='auto')
reduce_lr = ReduceLROnPlateau(monitor='val_acc', factor=0.1, patience=2, min_lr=0.000001, verbose=1, min_delta=0.001)

callbacks_list = [checkpointer, early, reduce_lr]

 

It’s time to train the model. Training logs are shown for each epoch. Note that the number of epochs is only 3, you may increase it. It’s less because later we will fine tune all the layers again with more epochs.

keras.backend.clear_session()
model = get_model()

model.fit_generator(train_gen,
                      steps_per_epoch=nb_train_samples//BATCH_SIZE, 
                      epochs=3, 
                      verbose=1, 
                      callbacks=callbacks_list, 
                      validation_data=valid_gen, 
                      validation_steps=nb_validation_samples//BATCH_SIZE,
                      max_queue_size=20, 
                      shuffle=True,
                      workers=4)

 

Now, since the added dense layers are trained for 3 epochs (or less depending on callbacks), its time to fine tune the whole model with a smaller learning rate, so we unfreeze the whole model.

for l in model.layers[:-1]:
    l.trainable = True

sgd = keras.optimizers.SGD(lr=0.0001, momentum=0.9)
model.compile(optimizer=sgd, loss=losses.categorical_crossentropy, metrics=['acc'])
model.summary()

model.fit_generator(train_gen,
steps_per_epoch=nb_train_samples//BATCH_SIZE,
epochs=200,
verbose=1,
callbacks=callbacks_list,
validation_data=valid_gen,
validation_steps=nb_validation_samples//BATCH_SIZE,
max_queue_size=20,
shuffle=True,
workers=4)

 

Now, we have the trained model, let’s save the weights for predicting on new images in the same fashion_mnist folder.

model.save('../../fashion_mnist/inception_resnet_v2_retrain_model.h5')

Running code on SAP Leonardo ML Foundation

Save all of the above training code as training.py in a folder code in the current directory. Also, create a yaml file by name fashion_mnist.yaml having following contents in the current directory:

job:
  name: "train-fashion-mnist"
  execution:
    image: "tensorflow/tensorflow:1.5.0-gpu"
    command: "pip install keras --upgrade && python training.py"
    completionTime: "10"
    resources:
      cpus: 1
      memory: 10000
      gpus: 1

Now, we have to upload the code folder having training.py and fashion_mnist.yaml to SAP Leonardo ML foundation as a job. As soon as the job is submitted it starts to run and you can also see the logs of the process. To upload, run in command prompt from the current directory:

cf sapml job submit -f fashion_mnist.yaml code

Predicting on new images

All we have to do now is to load the saved model and start predicting, making sure that we apply the same preprocessing steps as we did on the training dataset. But first, let’s download test data from the same place (if you haven’t done yet). It’s a zip, extracting it will give you a folder names test. Make another folder testing and put this test folder in the testing folder (we are doing this as this is how keras needs for prediction). Upload testing folder to SAP Leonardo ML foundation.

cf sapml fs put testing fashion_mnist/

Let’s start. Make a new python file prediction.py and write following code in it. After running this code, a final file having predictions for all the test images named submit.csv will get generated in the fashion_mnist folder in SAP Leonardo ML foundation.

## import libraries
import numpy as np
import pandas as pd

import os

import keras
from keras.preprocessing.image import ImageDataGenerator
from keras.models import load_model
from keras.preprocessing import image
from keras.applications.inception_resnet_v2 import preprocess_input

img_width, img_height = 28, 28
test_data_dir = '../../fashion_mnist/testing/'

datagen = ImageDataGenerator(rescale=1./255, preprocessing_function=preprocess_input)

model = load_model('../../fashion_mnist/inception_resnet_v2_retrain_model.h5')

generator = datagen.flow_from_directory(
        test_data_dir,
        target_size=(img_width, img_height),
        batch_size=128,
        seed=27,
        class_mode=None,  # only data, no labels
        shuffle=False)  # keep data in same order as labels
		
img_names = generator.filenames

probs = model.predict_generator(generator, verbose=1)

predictions = np.argmax(probs, axis=1)

df = pd.DataFrame(columns=['Image-Id', 'Category'])
df['id'] = img_names
df['label'] = predictions
df.to_csv('../../fashion_mnist/submit.csv', index=False)

The process to run the prediction is same as training:

  1. Put prediction.py in the same code folder.
  2. Change the yaml file, this should be the new content:
job:
  name: "predict-fashion-mnist"
  execution:
    image: "tensorflow/tensorflow:1.5.0-gpu"
    command: "pip install keras --upgrade && python prediction.py"
    completionTime: "5"
    resources:
      cpus: 1
      memory: 10000
      gpus: 1

3. Upload the prediction job:

cf sapml job submit -f fashion_mnist.yaml code

4. You can download the submit.csv file to your local system as well.

cf sapml fs get fashion_mnist/submit.csv .

 

Through this article:

  1. Doing basic image classification using keras and python.
  2. Uploading and downloading data from SAP Leonardo ML foundation.
  3. Running programs as jobs on SAP Leonardo ML foundation using Train Your Own Model (TYOM).
2 Comments
You must be Logged on to comment or reply to a post.
  • Hi, interesting article.

     

    What performance can you get out running this in the SAP Cloud Platform using their API? How many images can be classified per second?

     

    Best,

    Julian

    • Hi Julian, glad that you found the article interesting. The performance and number of images classified per second will hugely depend on:

       

      1. Image size, here images are very small 28*28, so it’s super fast.
      2. Model architecture, here I used inception_resnet_v2 just for an experiment, some different architecture may perform poor or better. You should experiment with a couple of architectures that come with keras already.
      3. The resources defined in the yaml file. You may use a different image, which gives you a very powerful GPU, also increase/decrease memory and number of CPUs in the yaml file. This all impacts the performance.

       

      Regards,

      Deepak