SAP Kyma: Replicating Real-time survey response from Qualtrics to SAP HANA Cloud using SAP Kyma as webhook

In this blog post, am going to cover on how to use SAP Kyma as a webhook to replicate real time survey responses from Qualtrics to SAP HANA Cloud. Also, to make it interesting am going to deploy the same business logic in Google Cloud App engine and use it as a webhook for a Qualtrics survey. Let us compare the deployments in both the environments. If you are interested in the Google Cloud Platform App Engine deployment, please refer this blog

And hey before you go through the blog, grab a cup of coffee 🙂 Its long but you will be able to understand the end-to-end scenario & complete the hands-on !

Consider this architecture which I have already covered the end-end-end use case  in a Discovery mission, and you can check it here: Visualize employee sentiments leveraging real-time analytics

Option1 :Cross Application Analytics using the SAP Business Technology Platform & SAP Kyma based Response Replication

  In this blog, am going to cover specifically the steps 1 to 6. The steps include

1. User completing the Qualtrics Survey.
2. Validating the Python Code using Qualtrics APIs and validating using Googe Colab
3. Containerize the script using Docker and making it available in public repository using Docker Hub.
4. Deploying the image in SAP Kyma
5. Exposing it as API for consuming it in web applications
6. Setting up the API URL as webhook in Qualtrics Survey and replicating the response in SAP HANA Cloud.

Let’s look into these steps in detail. You can refer the repo to access the python code for validating in Google Colab as well as for the Docker Deployment. We have added the deployment sequence in the Read.me file for your reference.

Step1: Importing Qualtrics Survey

If you already have a Qualtrics account [ trial or licensed], you should be able to download the survey “Employee_Comfort_Survey” from the GitHub repo. You can import the survey into your Qualtrics account, published and distributed the same to end users for completing the survey.  The survey has 6 Questions with selecting some options related to working from home and the final questions with you providing the comments about “working from home”.

Or you could also join the mission as mentioned earlier in the blog and we can provide you the survey link to share it with end users. You can share the URL with couple of users to complete the survey.

Please note that to use the Qualtrics APIs which is provided as part of the python code, you need to have a valid Qualtrics account. You cannot access APIs using the trial account. So, if it makes things easier contact us once you sign up for the discovery mission and we will provide the API keys, Survey IDs and Data center ID.  Make sure you complete the Step 1 before proceeding with step 1A. We need some test survey data for code validation.

Step 2: Code Validation

In order to validate the code, you need Anonymous or User Specific Survey link, Survey ID and API tokens for Qualtrics.  As mentioned step1, please reach out to Mission Experts to get access for survey and tokens or use your Qualtrics Account. And you would need access to SAP HANA Cloud trial account to replicate the survey response into a Database.  Please follow the steps mentioned in this link to sign up for SAP HANA Cloud trial  https://developers.sap.com/tutorials/hana-cloud-mission-trial-1.html

Once you have created the trial account, please download the schema  and import it in your SAP HANA Cloud trial account.

I have explained the steps in detail in this video . Or you can follow the steps in this blog too.

Once you have the necessary details from mission experts, navigate to https://colab.research.google.com/  with your google account to access the online Jupyter tool .  Please select the tab GitHub and provide the repository URL as https://github.com/IDGCOENA/UCR2802_SAPKyma

Please make sure the Repository is same as mentioned in both the lines and you will get access to UCR2802.ipynb.  Select “UCR2802.ipynb” and chose “New Notebook”

Colab will open the python script in a new notebook.

As mentioned in the Read.me file, you have to provide credentials and tokens in the code.

1. Provide the Parameters in the lines 16 – 19 [HANA Cloud DB Username, Password , Qualtrics Survey ID , Qualtrics API Token] with your HANA Cloud Trial Logon details . And for Qualtrics tokens and IDs, please reach out to Mission Support.
2. Provide the HANA Cloud host URL at line 86.
3. Provide the Qualtrics Data center at line 165 – Please reach out to Mission Support to request the same.
4. Save the file after updating the necessary credentials and tokens.

Please request the survey URL(anonymous or User Specific) from the support team to share with your team to take the survey.  Once you have the URL, please complete the survey.

Navigate back to notebook and Select “Run all ” to execute the code.

Once the execution is successful, you should see the following log in the bottom of the screen.

To confirm the survey response data is replicated into HANA Cloud, please navigate to your BTP schema using the DB explorer.

And select “Open Data” for the table EMPLOYEE_COMFORT_SURVEY_USERINFO. You should see the response data of the surveys.

Step3: Containerize the Application using Docker and deploy it in public repository

If you do not have a docker ID , please sign up  here https://docs.docker.com/docker-id/.

Then download the docker desktop from this link  https://www.docker.com/products/docker-desktop

I have explained the steps in detail in this video .

Or you can follow the steps in this blog too.

Once you have downloaded Docker Desktop, please use the docker ID you have just created  and login in to docker desktop

Download or clone the repository to your local desktop. If you have downloaded, extract the zip file and save it in a separate folder.

Navigate to the folder and remove the UCR2802.ipynb file and the Qualtrics survey. The folder should have the following files

Now you have to make adjustments to the files main.py and deployment.yaml.

In the main.py, you will adjust the python code similar to Step1: Importing Qualtrics Survey.The difference in main.py is I have used the micro web framework flask to expose the code as web service.

Changes in main.py file

1. Provide the Parameters in the lines 18 – 22 [HANA Cloud DB Username, Password , Qualtrics Survey ID , Qualtrics API Token] with your HANA Cloud Trial Logon details . And for Qualtrics tokens and IDs, please reach out to Mission Support.
2. Provide the HANA Cloud host URL at line 99.
3. Provide the Qualtrics Data center at line 193 – Please reach out to Mission Support to request the same.

Save the file after updating the necessary credentials and tokens.

Changes in deployment.yaml file

Change the line 31 with your dockerid/<anyimagename>.

In my case, my dockerID is sapkyma and  chose the imagename as qresponse1

Once completing all the changes, navigate to the same folder from Command Prompt

Execute    docker build .  -t    <your dockerid>/<any image name>

Then you should see the build finished with all dependencies.

And it should be visible from your docker desktop now . You will see qresponse2 in my case even though the yaml had qresponse1.  I had both the images build as I was using the second image while recording the video.

Then we need to push the image to public repository  “docker hub”

Execute    docker push    <your dockerid>/<any image name>

Once it is mounted, you should be able to view from Docker hub

Step4: Deployment in SAP Kyma

I assume that you have already signed up for SAP Kyma free trial. If not please sign up by following this link https://developers.sap.com/tutorials/cp-kyma-getting-started.html

Upon completion of step3, we have the image available in public repository with Docker Hub. We will use the modified Deployment.yaml file [from your local desktop] to deploy the image in SAP Kyma.

Navigate to your trial subaccount and select “Kyma Environment”

Then Select “Link to Dashboard” which will take you to default namespace screen

Select default Namespace and it will take you to the overview screen.

Select the “Deploy New Workload” option and select “Upload Yaml” function . You can drag the deployment.yaml  from your local folder and upload it here.  Please make sure you to use the modified deployment.yaml  from step 3[as shown below] .

Upload the deployment.yaml file as shown below.  The yaml file has the public repo based on your docker hub and Kyma will deploy the container accordingly.

Once you select deploy, you should see the app “main” deployed and running.

Step 5: Exposing it as API for consuming it in web applications

Next is to expose the app as API. So under Discovery And Network, click API Rules and Select “Create API rule”

Provide any name and hostname . From the service select the app main(port:5050) . Leave the access strategies as such.  The screenshot shows main2 but in your case when you deploy the yaml, it will be displayed as main(port: 5050).

Once you complete the step, you should see the Host URL for your app.

Click on the URL for main app , and you should see this Message on web page. Make sure your HANA Cloud is still running and you have already completed the survey using the Survey URL.

To check if the app replicates data into SAP HANA Cloud ,  you could use the Database Explorer for HANA Cloud and execute the two SQL statements

TRUNCATE TABLE “BTP”.”EMPLOYEE_COMFORT_SURVEY_QUESTIONS”;

TRUNCATE TABLE “BTP”.”EMPLOYEE_COMFORT_SURVEY_USERINFO”;

Now there are no entries in these tables. Use the anonymous or user specific survey link to complete the surveys.

After completing the survey, refresh  the Kyma URL  web page again .  You should see the same message as “Survey Response has been Successfully replicated into HANA Cloud” . And you query the same tables again

SELECT * FROM “BTP”.”EMPLOYEE_COMFORT_SURVEY_QUESTIONS”;

This query should display all the survey questions.

SELECT * FROM “BTP”.”EMPLOYEE_COMFORT_SURVEY_USERINFO”;

This query should display all the survey response.

Congrats, you have completed the Kyma Deployment.  Copy this URL for adding this as webhook to Qualtrics survey.

Step6: Setting up the API URL as webhook in Qualtrics Survey and replicating the response in SAP HANA Cloud.

Login to you Qualtrics Account and select the Survey. Once the survey is loaded, select the Workflow tab.

In the workflow tab, create a new workflow “Started by an event”

Select the workflow for the ” Survey Response”

Once you select the workflow, select the option for newly created responses

Once you select “Finish”, Select Add task and select the task as “Web Service”

As we are testing the webhook, you can use Non-authenticated connection

Select the “Non-authenticated” option and provide the Exposed API URL from Kyma as web hook.

Once saved, share the survey URL with users to complete the survey. Once they have completed the survey, the response will be automatically replicated into SAP HANA Cloud. Hope this blog helps you in understanding the end-to-end scenario for setting up surveys in Qualtrics and replicating the real time response using SAP Kyma . For the deployment using Google Cloud Platform App Engine, please refer this blog

I would like to thank Jamie Cawley  for his interesting blogs on Kyma & helping me with deployment issues , Carlos Roggan  for explaining docker set up with a meow example 🙂  &  Kevin Li  for his contributions to python developments.

Please take a look into the mission and share feedback on your BTP journey. We are excited to hear your feedback. Please feel free to comment and also reach out to me if you have issues in accessing the Git Repositories .