SAP Kyma: Replicating Real-time survey response from Qualtrics to SAP HANA Cloud using SAP Kyma as webhook
In this blog post, am going to cover on how to use SAP Kyma as a webhook to replicate real time survey responses from Qualtrics to SAP HANA Cloud. Also, to make it interesting am going to deploy the same business logic in Google Cloud App engine and use it as a webhook for a Qualtrics survey. Let us compare the deployments in both the environments. If you are interested in the Google Cloud Platform App Engine deployment, please refer this blog
And hey before you go through the blog, grab a cup of coffee 🙂 Its long but you will be able to understand the end-to-end scenario & complete the hands-on !
Consider this architecture which I have already covered the end-end-end use case in a Discovery mission, and you can check it here: Visualize employee sentiments leveraging real-time analytics
Option1 :Cross Application Analytics using the SAP Business Technology Platform & SAP Kyma based Response Replication
In this blog, am going to cover specifically the steps 1 to 6. The steps include
- User completing the Qualtrics Survey.
- Validating the Python Code using Qualtrics APIs and validating using Googe Colab
- Containerize the script using Docker and making it available in public repository using Docker Hub.
- Deploying the image in SAP Kyma
- Exposing it as API for consuming it in web applications
- Setting up the API URL as webhook in Qualtrics Survey and replicating the response in SAP HANA Cloud.
Let’s look into these steps in detail. You can refer the repo to access the python code for validating in Google Colab as well as for the Docker Deployment. We have added the deployment sequence in the Read.me file for your reference.
Step1: Importing Qualtrics Survey
If you already have a Qualtrics account [ trial or licensed], you should be able to download the survey “Employee_Comfort_Survey” from the GitHub repo. You can import the survey into your Qualtrics account, published and distributed the same to end users for completing the survey. The survey has 6 Questions with selecting some options related to working from home and the final questions with you providing the comments about “working from home”.
Or you could also join the mission as mentioned earlier in the blog and we can provide you the survey link to share it with end users. You can share the URL with couple of users to complete the survey.
Please note that to use the Qualtrics APIs which is provided as part of the python code, you need to have a valid Qualtrics account. You cannot access APIs using the trial account. So, if it makes things easier contact us once you sign up for the discovery mission and we will provide the API keys, Survey IDs and Data center ID. Make sure you complete the Step 1 before proceeding with step 1A. We need some test survey data for code validation.
Step 2: Code Validation
In order to validate the code, you need Anonymous or User Specific Survey link, Survey ID and API tokens for Qualtrics. As mentioned step1, please reach out to Mission Experts to get access for survey and tokens or use your Qualtrics Account. And you would need access to SAP HANA Cloud trial account to replicate the survey response into a Database. Please follow the steps mentioned in this link to sign up for SAP HANA Cloud trial https://developers.sap.com/tutorials/hana-cloud-mission-trial-1.html
Once you have created the trial account, please download the schema and import it in your SAP HANA Cloud trial account.
I have explained the steps in detail in this video . Or you can follow the steps in this blog too.
Once you have the necessary details from mission experts, navigate to https://colab.research.google.com/ with your google account to access the online Jupyter tool . Please select the tab GitHub and provide the repository URL as https://github.com/IDGCOENA/UCR2802_SAPKyma
Please make sure the Repository is same as mentioned in both the lines and you will get access to UCR2802.ipynb. Select “UCR2802.ipynb” and chose “New Notebook”
Colab will open the python script in a new notebook.
As mentioned in the Read.me file, you have to provide credentials and tokens in the code.
- Provide the Parameters in the lines 16 – 19 [HANA Cloud DB Username, Password , Qualtrics Survey ID , Qualtrics API Token] with your HANA Cloud Trial Logon details . And for Qualtrics tokens and IDs, please reach out to Mission Support.
- Provide the HANA Cloud host URL at line 86.
- Provide the Qualtrics Data center at line 165 – Please reach out to Mission Support to request the same.
- Save the file after updating the necessary credentials and tokens.
Please request the survey URL(anonymous or User Specific) from the support team to share with your team to take the survey. Once you have the URL, please complete the survey.
Navigate back to notebook and Select “Run all ” to execute the code.
Once the execution is successful, you should see the following log in the bottom of the screen.
To confirm the survey response data is replicated into HANA Cloud, please navigate to your BTP schema using the DB explorer.
And select “Open Data” for the table EMPLOYEE_COMFORT_SURVEY_USERINFO. You should see the response data of the surveys.
Step3: Containerize the Application using Docker and deploy it in public repository
If you do not have a docker ID , please sign up here https://docs.docker.com/docker-id/.
Then download the docker desktop from this link https://www.docker.com/products/docker-desktop
I have explained the steps in detail in this video .
Or you can follow the steps in this blog too.
Once you have downloaded Docker Desktop, please use the docker ID you have just created and login in to docker desktop
Download or clone the repository to your local desktop. If you have downloaded, extract the zip file and save it in a separate folder.
Navigate to the folder and remove the UCR2802.ipynb file and the Qualtrics survey. The folder should have the following files
Now you have to make adjustments to the files main.py and deployment.yaml.
In the main.py, you will adjust the python code similar to Step1: Importing Qualtrics Survey.The difference in main.py is I have used the micro web framework flask to expose the code as web service.
Changes in main.py file
- Provide the Parameters in the lines 18 – 22 [HANA Cloud DB Username, Password , Qualtrics Survey ID , Qualtrics API Token] with your HANA Cloud Trial Logon details . And for Qualtrics tokens and IDs, please reach out to Mission Support.
- Provide the HANA Cloud host URL at line 99.
- Provide the Qualtrics Data center at line 193 – Please reach out to Mission Support to request the same.
Save the file after updating the necessary credentials and tokens.
Changes in deployment.yaml file
Change the line 31 with your dockerid/<anyimagename>.
In my case, my dockerID is sapkyma and chose the imagename as qresponse1
Once completing all the changes, navigate to the same folder from Command Prompt
Execute docker build . -t <your dockerid>/<any image name>
Then you should see the build finished with all dependencies.
And it should be visible from your docker desktop now . You will see qresponse2 in my case even though the yaml had qresponse1. I had both the images build as I was using the second image while recording the video.
Then we need to push the image to public repository “docker hub”
Execute docker push <your dockerid>/<any image name>
Once it is mounted, you should be able to view from Docker hub
Step4: Deployment in SAP Kyma
I assume that you have already signed up for SAP Kyma free trial. If not please sign up by following this link https://developers.sap.com/tutorials/cp-kyma-getting-started.html
Upon completion of step3, we have the image available in public repository with Docker Hub. We will use the modified Deployment.yaml file [from your local desktop] to deploy the image in SAP Kyma.
Navigate to your trial subaccount and select “Kyma Environment”
Then Select “Link to Dashboard” which will take you to default namespace screen
Select default Namespace and it will take you to the overview screen.
Select the “Deploy New Workload” option and select “Upload Yaml” function . You can drag the deployment.yaml from your local folder and upload it here. Please make sure you to use the modified deployment.yaml from step 3[as shown below] .
Upload the deployment.yaml file as shown below. The yaml file has the public repo based on your docker hub and Kyma will deploy the container accordingly.
Once you select deploy, you should see the app “main” deployed and running.
Step 5: Exposing it as API for consuming it in web applications
Next is to expose the app as API. So under Discovery And Network, click API Rules and Select “Create API rule”
Provide any name and hostname . From the service select the app main(port:5050) . Leave the access strategies as such. The screenshot shows main2 but in your case when you deploy the yaml, it will be displayed as main(port: 5050).
Once you complete the step, you should see the Host URL for your app.
Click on the URL for main app , and you should see this Message on web page. Make sure your HANA Cloud is still running and you have already completed the survey using the Survey URL.
To check if the app replicates data into SAP HANA Cloud , you could use the Database Explorer for HANA Cloud and execute the two SQL statements
TRUNCATE TABLE “BTP”.”EMPLOYEE_COMFORT_SURVEY_QUESTIONS”;
TRUNCATE TABLE “BTP”.”EMPLOYEE_COMFORT_SURVEY_USERINFO”;
Now there are no entries in these tables. Use the anonymous or user specific survey link to complete the surveys.
After completing the survey, refresh the Kyma URL web page again . You should see the same message as “Survey Response has been Successfully replicated into HANA Cloud” . And you query the same tables again
SELECT * FROM “BTP”.”EMPLOYEE_COMFORT_SURVEY_QUESTIONS”;
This query should display all the survey questions.
SELECT * FROM “BTP”.”EMPLOYEE_COMFORT_SURVEY_USERINFO”;
This query should display all the survey response.
Congrats, you have completed the Kyma Deployment. Copy this URL for adding this as webhook to Qualtrics survey.
Step6: Setting up the API URL as webhook in Qualtrics Survey and replicating the response in SAP HANA Cloud.
Login to you Qualtrics Account and select the Survey. Once the survey is loaded, select the Workflow tab.
In the workflow tab, create a new workflow “Started by an event”
Select the workflow for the ” Survey Response”
Once you select the workflow, select the option for newly created responses
Once you select “Finish”, Select Add task and select the task as “Web Service”
As we are testing the webhook, you can use Non-authenticated connection
Select the “Non-authenticated” option and provide the Exposed API URL from Kyma as web hook.
Once saved, share the survey URL with users to complete the survey. Once they have completed the survey, the response will be automatically replicated into SAP HANA Cloud. Hope this blog helps you in understanding the end-to-end scenario for setting up surveys in Qualtrics and replicating the real time response using SAP Kyma . For the deployment using Google Cloud Platform App Engine, please refer this blog
I would like to thank Jamie Cawley for his interesting blogs on Kyma & helping me with deployment issues , Carlos Roggan for explaining docker set up with a meow example 🙂 & Kevin Li for his contributions to python developments.
Please take a look into the mission and share feedback on your BTP journey. We are excited to hear your feedback. Please feel free to comment and also reach out to me if you have issues in accessing the Git Repositories .