Machine Learning (ML) Retraining Pipeline

[TripathiRahul]

Do you want to develop an ML app with the ability to continuously improve your model using retraining (i.e., re-running the training process on new set of data.)?

This blog post will help you understand the steps required to set up the entire retraining pipeline, including:

• Uploading the initial training dataset to the Google Cloud Platform bucket.


• Training the model on the Google Cloud Platform (GCP) and deploying it to BTP Mobile Services Client Resources.


• Setting up the app.


• Capturing success and uploading the image for retraining.

Before beginning, you should already have a basic knowledge of SAP BTP Mobile Services, GCP and Firebase.

The following diagram illustrates the entire flow for an image recognition app with a retraining pipeline

Basic Outline of the Pipeline

Steps

Upload the initial training dataset

Collect the initial training dataset and upload it directly to the GCP bucket.

Train the model on GCP and deploy it to BTP Mobile Services Client Resources

Download the dataset from the GCP bucket to the VM instance on GCP

#download_dataset.py



from google.cloud import storage



storage_client = storage.Client()



bucket = storage_client.bucket("<project-id>.appspot.com")

blob = bucket.blob("<dataset_name>.zip")

blob.download_to_filename("<download_dataset_name>.zip")

 

Perform training on the downloaded dataset and get the MLModel

Use Turi Create to train your model and export the model.

# training.py



#after training, export the model to core ml model

<Trained_Model_Name>.export_coreml('<CoreML_Model_Name>.mlmodel')

 

Deploy the Core ML model to BTP Mobile Services Client Resources

Do the following:

1. Do a PUT of the Client Resources Service as described here and then export your collection.


2. Upload the exported collection to your GCP VM instance.


3. Install Newman (the command line interface for Postman) on GCP.


4. To deploy this model to SAP BTP Mobile Services Client Resources, run the following command in the terminal:

newman run <collection_name>.json

Going forward, you just need to change the version number for deployment in the collection file.

 

Setting up the app

 

To set up the app, do the following:

1. Add Firebase to your iOS app.


2. If you want to keep track of success rate, enable analytics.


3. Pick the latest version of the Core ML Model from SAP BTP Mobile Services Client Resources using SAPMLModelManager and use it in your app.

After following these steps, you can perform prediction request using the downloaded ML Model from SAP BTP Mobile Services Client Resources and the app is now ready to capture success.

 

Capture success and upload the image for retraining

 

The final step in completing the pipeline is to capture the success and upload it to the GCP bucket (success here means when any step/action performed after getting the predicted result.)

If the image is a success, upload it and use its predicted label to define the GCP bucket path.

if let imageData = <captured_image>.jpegData(compressionQuality: 1){

            let storage = Storage.storage()

            storage.reference().child("<dataset_name>/\((self.user?.label!)!)/<image_name>.png").putData(imageData, metadata: nil){

                (_, err) in

                if let err = err{

                    print("an error occured- \(err.localizedDescription)")

                } else {

                    print("Image Uplaoded Successfully")

                }

            }

        } else {

            print("couldn't wnwrap image")

        }

 

Capture the success count for analytics

Record the success count for the current version of the model.

Analytics.setDefaultEventParameters([

  "success_rate": <success_count_value>,

  "curr_version": <curr_version_number>

])

 

 

Conclusion

Using the information contained in this blog post, you should now be able to create an ML app with improved learning and model capabilities i.e., you can use the data captured on device for prediction to improve the model by triggering the retraining pipeline.

This is my first blog post and your suggestions/feedbacks are highly appreciated. There are few more articles on the way. Please follow my profile for the new blog posts related to latest features provided by SAP.

Do you have any further comments/questions related to this topic? Do share them in the comments sections below without any hesitation.

Note – All the images are created by me and free to use/share.