UPDATE March 15, 2016
Thank You for you interest. Due to recent changes in the features offered in the SAP HANA Cloud Platform Trial Edition, some parts in the scenario below no longer reflect the current situation.
We are considering to update this playlist in the near future but do not have an estimated timeframe as yet.
Last month, Philip Mugglestone of the SAP HANA Academy recorded a new series of tutorial videos in which he shows how you can build an end-to-end real-time social media analytics application using the SAP HANA Cloud Platform.
The playlist of the series contains 37 videos in total, total duration almost 5 hours. The development project and application is called Live3.
The source data comes at no cost from a social media feed. The services are provided by the SAP HANA Cloud Platform developer edition, a free to use Platform as a Service (PaaS). The developer tools are available for the Windows, Linux and MacOS platforms, at no charge and this concerns two SAP plugins for the Eclipse IDE, the SAP HCP SDK and the open source Google Chrome runtime, Node.js. There are no prerequisites. you can start from scratch.
YouTube Playlist: Live3 – SAP HANA Cloud Platform for Advanced Real-Time Social Media Analytics – YouTube
All content from this project is available on Github https://github.com/saphanaacademy/Live3HCP
Project Description
This enablement offering helps you get started with the SAP HANA Cloud Platform developer edition by walking through a step-by-step project to build a mobile application using real-time data feeds that combines sentiment analysis with predictive analytics whilst exploiting native development features of the SAP HANA Cloud Platform.
The Live3 project allows the CMO to better understand social network sentiment relating to his company, services, or products in order to identify those in a position of influence that publish negative sentiment – so that this segment they can be recruited into a specialized education program. The insight evolves in real-time as new data is posted to social media sites.
Live3 a native SAP HANA application built using the SAP HANA Cloud Platform – leveraging real-time data feeds from social media, data modeling in SAP HANA tools for Eclipse, native SAP HANA text analytics for sentiment analysis as well as native SAP HANA clustering for segmentation, SAP HANA extended application services (XS), OData, and SAP Web IDE to build an HTML5 application using SAPUI5. It can also easily be deployed as a native mobile app.
Whether you are new to SAP HANA Cloud Platform, or just looking to expand your knowledge, this is a great opportunity to gain hands-on experience with the SAP HANA Cloud Platform.
1. Introduction
In the first video Philip discusses the components of the scenario to analyze streaming Twitter data using the SAP HANA Cloud Platform developer edition.
https://youtube.com/watch?v=la9V2hAeuK8
See also Tom’s post: [SAP HANA Academy] Live3: Introduction
2. Register for SAP HANA Cloud Platform Trial
In this video we will register for the SAP HANA Cloud Platform developer edition and create our first SAP HANA instance.
https://youtube.com/watch?v=ld4_V41QVfk
See also Tom’s post: [SAP HANA Academy] Live3: Register for SAP HANA Cloud Platform Trial
3. Install SAP HANA Cloud Platform Tools
In this video after installing the JDK for Java we will download and install SAP HANA Cloud Platform tools including the Web SDK and Eclipse Luna. We will also install SAP HANA Cloud Platform Tools and SAP HANA Tools into the Eclipse environment.
https://youtube.com/watch?v=Hh9ElqQarxU
See also Tom’s post:[SAP HANA Academy] Live3: Install SAP HANA Cloud Platform Tools
4. Add Cloud System
In this video we will add a cloud system from the Eclipse SAP HANA Administration perspective in order to connect to our SAP HANA Cloud Platform developer edition instance.
https://youtube.com/watch?v=mThzHh06iCk
See also Tom’s post: [SAP HANA Academy] Live3: Add Cloud System
5. Identify & Set Application Schema
In this video we will identify and set our application schema in the Eclipse SAPHANA Administration perspective.
https://youtube.com/watch?v=arXF02oLWC0
See also Tom’s post:[SAP HANA Academy] Live3: Identify & Set Application Schema
6. Create Tweets Table
In this video we will create a new SAP HANA database table to contain tweets that we are interested in analyzing further. The data to be loaded will be obtained in real-time via the Twitter API.
https://youtube.com/watch?v=y50lnzHCpP8
See also Tom’s post: [SAP HANA Academy] Live3: Create Tweets Table
7. Setup Twitter
In this video we will setup Twitter as a pre-requisite to using the Twitter API. After reviewing the Twitter API documentation, we will create a Twitter application and generate the consumer and access token keys required to authenticate to the Twitter streaming API.
https://youtube.com/watch?v=YI85ZECYsZo
See also Tom’s blog post:[SAP HANA Academy] Live3: Setup Twitter
8. Setup node.js
In this video we will install node.js.
https://youtube.com/watch?v=gM4SGBxXJeA
See also Tom’s post: [SAP HANA Academy] Live3: Setup node.js
9. node.js Application
In this video we will review the node.js application that creates a local web server which accesses the Twitter streaming API, receives and parses tweets, then loads them into the Tweets table off our SAP HANA database.
https://youtube.com/watch?v=VCWMJoDEmWY
See also Tom’s blog post: [SAP HANA Academy] Live3: node.js Application
10. Setup HCP Tunnel
In this video we will configure and start a tunnel that enables our Node.JS application to communicate directly with our SAP HANA Cloud Platform trial instance.
https://youtube.com/watch?v=pjZiWDjdrzI
See also Tom’s blog post: [SAP HANA Academy] Live3: Setup HCP Tunnel
11. Load Tweets
In this video we will use the Node.JS application to query the Twitter streaming API and load desired tweet information into our SAP HANA instance within the SAP HANA Cloud Platform.
https://youtube.com/watch?v=O_dQzkE8Xcc
See also Tom’s blog post: [SAP HANA Academy] Live3: Load Tweets
12. Text Analysis
In this video we will perform text analysis using the native capabilities provides by SAP HANA in order to identify the sentiment of persons sending tweets – for example strong positive, weak positive, weak negative, or strong negative sentiment.
https://youtube.com/watch?v=Dmo-ekJUe4M
See also Tom’s blog post: [SAP HANA Academy] Live3: Text Analysis
13. Influence & Stance View
In this video we will create a view containing scores for each tweeter based on influence and stance (attitude). Influence reflects how influential the tweeter is – for example # retweets and replies received by a user compared to retweets and replies sent by the user. Stance reflects whether the user is generally positive or negative based on the sentiment derived during text analysis. The influence and stance scores will be input to the upcoming clustering (or segmentation) step.
https://youtube.com/watch?v=PEt25w9EjGI
14. Explain Clustering
In this video we will learn more about the SAP HANA predictive analysis library (or PAL) and how it can be used to cluster (or segment) tweeters based on their influence and stance scores. We will review the k-means clustering algorithm.
https://youtube.com/watch?v=_9s_hIuiMjk
15. Perform Clustering
In this video we will review and run the SQL Script necessary to setup and execute k-means clustering analysis using the SAP HANA predictive analysis library. We will also create some views to be accessed subsequently via web services.
https://youtube.com/watch?v=xEGX64AUDiM
16. SAP HANA Web-based Development Workbench
In this video we will introduce the SAP HANA Web-based Development Workbench and create our live3 web services project.
https://youtube.com/watch?v=cNmtNT-1EnQ
17. Web Services – Authorizations
In the this video we will configure authorizations for web services. Application privileges and user roles will be defined.
https://youtube.com/watch?v=qs061nzBdyU
18. Web Services – Setup OData
In this video we will introduce what OData is and set up an OData service as part of our live3 web services project.
https://youtube.com/watch?v=KjsYgAmUwHY
19. Web Services – Using OData
In this video we will explore the OData web services created previously and make some example queries.
https://youtube.com/watch?v=0MQgsMsSZjc
20. Web Services – Setup XSJS
In this video, we will add server-side scripting capabilities to our web services project. This will allow us to configure actions to refresh the clustering and “reset” the database.
https://youtube.com/watch?v=ZdKk_VXCwUw
21. Web Services – Debugging XSJS
In this video we will see how server-side scripting applications can easily be debugged using the SAP HANA Web-based Development Workbench.
https://youtube.com/watch?v=24GLegisPqY
22. Web Services – Authentication
In this video we will understand why a “proxy” authentication server is needed to access your SAP HANA Cloud Platform web services from a SAP HANA Cloud HTML5 application.
https://youtube.com/watch?v=Kaun6PPupzA
23. Web Services – Authentication Setup Proxy
In this video we will setup the “proxy” authentication server.
https://youtube.com/watch?v=fhtbO532DiI
24. Setup SAP HANA Cloud Connector
In this video we will install and configure the SAP HANA Cloud Connector.
https://youtube.com/watch?v=jqGfXL9Vz00
25. Setup SAP HANA Cloud Connector – Resources
In this video we will define SAP HANA Cloud Connector resources in order to make back-end systems available to the SAP HANA Cloud Platform.
https://youtube.com/watch?v=_0gHqaf2Rvw
26. Setup SAP HANA Cloud Platform – Destinations
In this video we will define SAP HANA Cloud Platform “destinations” to reference back-end resources previously configured using the SAP HANA Cloud Connector.
https://youtube.com/watch?v=1_E7MOyCJ4k
27. Setup SAP Web IDE
In this video we will setup SAP Web IDE so that we can more easily develop our HTML5 application based on SAPUI5.
https://youtube.com/watch?v=jWM1i82qr5o
28. SAP Web IDE – Templates
In this video we will use SAP Web IDE to generate an SAPUI5 application driven by the OData services we created previously with SAP HANA XS. We will introduce and use the SAP Web IDE “Master Master Detail” template.
https://youtube.com/watch?v=myJhpRiLs10
29. SAP Web IDE – Deploy HTML5
In this video we will use SAP Web IDE to deploy our SAPUI5 project as a fully fledged SAP HANA Cloud Platform HTML5 application.
https://youtube.com/watch?v=0dhffxH0EFQ
30. SAP Web IDE – Code Review
In this video we will review the structure and code that was generated by the “Master Master Detail” template of SAP Web IDE.
https://youtube.com/watch?v=NSQJhQR8JU8
31. SAP Web IDE – Customize Master – List
In this video we will customize the code that was generated by the “Master Master Detail” template of SAP Web IDE. We will add additional attributes to the list in the master view.
https://youtube.com/watch?v=X98CIrgdmf8
32. SAP Web IDE – Customize Master – Toolbar
In this video we will customize the code that was generated by the “Master Master Detail” template of SAP Web IDE. We will add a Pull To Refresh control to the master view and an icon toolbar to the footer.
https://youtube.com/watch?v=-HVKAbwANY0
33. SAP Web IDE – Customize Master – Methods
In this video we will customize the code that was generatedby the “Master Master Detail” template of SAP Web IDE. We will modify the search method and add new methods for the toolbar icons.
https://youtube.com/watch?v=RNYK1HvnRW0
34. SAP Web IDE – Customize Master – Test
In this video we will customize the code that was generated by the “Master Master Detail” template of SAP Web IDE. We will configure and test the toolbar icons to perform clustering, reset, and to start/stop real-time loading of data via the Twitter API.
https://youtube.com/watch?v=sGVm5ENw7Hs
35. SAP Web IDE – Customize Master 2 & Detail
In this video we will customize the code that was generated by the “Master Master Detail” template of SAP Web IDE. We will add additional attributes to the list in the master2 view and a list of tweets to the detail view.
https://youtube.com/watch?v=AEyWk-iQVIc
36. SAP Web IDE – Customize Master 2 Controller
In this video, we will customize the code that was generated by the “Master Master Detail” template of SAP Web IDE. We will fix an issue in the master2 controller that causes the details view to be refreshed unnecessarily.
https://youtube.com/watch?v=EaRY9hTWiuM
37. SAP Web IDE – Commit Push HTML5
In this video we will use SAP Web IDE to deploy our SAPUI5 project as a fully fledged SAP HANA Cloud Platform HTML5 application. We will stage, commit and push our changes and activate a new app version.
https://youtube.com/watch?v=jZUY-cGJuuw
38. Test With Mobile Device
In this video we will test the completed application on an iPhone.
https://youtube.com/watch?v=i97PD-_e3rs
39. Java – Create Project
In this video we cover an alternative approach to loading Twitter data into SAP HANA data via Node.js. We will create a Java application. After setting up Eclipse for Java development with SAP HANA Cloud Platform, we will create the Java project then download the Java source code from Github and perform some initial configuration.
https://youtube.com/watch?v=d3ixBc6bW9s
40. Java – Souce Code Walkthrough
In this video, Mac Holden walks us through the source code of the Java project.
https://youtube.com/watch?v=z_-Ru-ygKes
41. Java – Deploy to HCP and Test
In this video, we configure appropriate HCP roles and data bindings, test the application, and load new tweets into SAP HANA.
https://youtube.com/watch?v=hIev1FeqR-k
42. Java – Update HTML5 App, Deploy and Test
In this video, we will create a HCP destination for our Java application and make the necessary source code changes to the HTML5 app with SAP Web IDE, then update the deployed version and verify that the end to end scenario
Hello Philip and thank you for these great series !
Unfortunately there is a problem on quering the tweets.
Although i enter "/do/start?track=%23SAPHANA%20" translating to #SAPHANA(space)
i get tweets that have nothing to do with this.
Especially from Greece. (where i live)
Is there any chance to me synced with my twitter account ?
What am i doing wrong?
Best regards,
Christoforos Verras
<edit>:
after lots of "debugging" i am now sure that it tracks also the tweets from the users i follow.
A solution might be to create a new account with no followers at all..
<edit>:
Hi Christoforos,
Thanks for the feedback.
Indeed, I'd recommend creating a new twitter account that doesn't follow anyone.
For the filtering, the Twitter Streaming API is documented here and explains some of the subleties of how the filtering is designed to work: https://dev.twitter.com/streaming/overview/request-parameters#track
Thanks,
Philip
Hi Philip ,
So we actually did a new Twitter account , without any followers and it works perfect.
Thank you so much !
best regards ,
Chris
Hi Philip,
From past couple of weeks I have been extensively working on HCP and building Iot related scenarios. I came across one for your scenario of reading the tweets, processing it in XS engine and displaying it in nice HTML5 app.
First of all I must say its an amazing video showcasing the end to end scenario and really helpful for the people working on HCP.
In the set of those 37 videos, there is a video where you explain the usage of HANA Cloud connector and why is it necessary to use CC.
I want to understand couple of things here where I need your confirmation and guidance.
1) I have an Odata provider which comes from on premise ECC system.
2) I have an HTML5 app deployed on HCP.
3) I need to now create a destination in HCP to backend ECC system. For this I mandatorily need to use Cloud connector right ?
4) In your example you have used a proxy to connect html5 and native hana xs using CC. But this is only applicable for trial instance right. In the actual environment we don't need to do that and HTML5 can directly connect to HANA XS via destination configuration?
Thanks in advance.
Hi Rahul ,
I have a question about the Twitter application.
Do you only track for the word you want, or did you ever try to search for previously published tweets ?
What I am trying to do is to use the search API instead of the streaming API.
Any help would be appreciated !
Best,
Christoforos Verras
Hi Christoforos
I only tried with streaming api
Regards,
Rahul
Hi Rahul,
Thanks for the nice feedback.
CC is needed when you have a back-end system that is not visible on the public internet i.e. behind some kind of firewall. CC runs on a machine behind the firewall and creates a "private tunnel" between that machine and your HCP account - this then means that HCP destinations can reach the back end system without having to make the back end system publicly visible on the internet. Make sense?
So if you have a back-end system that's behind a firewall, then yes use CC to enable HCP destinations to reach it. If however your back-end system is already visible on the public internet no need for CC - just point the HCP destinations to it directly.
Yes, the use of CC to connect to HCP HANA XS only applies to the free developer trial of HCP. In HCP productive systems, HCP destinations can connect & authenticate directly.
Thanks,
Philip
Thanks a lot Philip for clarifying this. I am trying this now with one of our ECC systems behind the firewall. Will get back in case of any issues
Regards
Rahul
Hi Philip
I have a question regarding setting up the HANA instance in HCP. We have currently set up a default instance. Can we set up a different HANA instance here. Why this question because we have Operational Process Intelligence running on a HANA box and through HANA XS application in HCP I want to push this data to a different HANA box where OPI is installed. Is it possible to do that?
Regards
Rahul
Hi Rahul,
If i understood your question correclty, HCP allows you to have only one instance.
Thanks,
Christoforos Verras
Hi Christoforos
Actually I was thinking to delete the default instance and create another one pointing to my HANA instance where OPInt is installed. What do you think? Will that work?
Regards
Rahul
Hi Rahul,
HCP only allows a single HANA instance in the developer trial. If you delete the one you currently have you can only then create a new empty one. The HCP starter edition in the production landscape allows multiple instances so might be better suited to your needs? Also, in the production landscape you won't need the hana-auth-proxy (see video 23) to connect the HTML5 app to the HANA instance.
Thanks,
Philip
hi expert,
I'm really got crazy about this step 11 "load tweets". this is the error log:
Error: getaddrinfo ENOTFOUND userstream.twitter.com userstream.twitter.com:443
at errnoException (dns.js:26:10)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:77:26)
everything is ok util I enter the http://localhost:8888/do/start?track=ausopen the page shows "start" while the cmd shows the error
PlZ help me !!!!!!!!
hi expert,
I'm really got crazy about this step 11 "load tweets". this is the error log:
Error: getaddrinfo ENOTFOUND userstream.twitter.com userstream.twitter.com:443
at errnoException (dns.js:26:10)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:77:26)
everything is ok util I enter the http://localhost:8888/do/start?track=ausopen the page shows "start" while the cmd shows the error
PlZ help me !!!!!!!!
Hi Joe,
This message sometimes appears when your node.js application is behind a firewall so it cannot reach the Twitter API. In that case it's necessary to declare the proxy server.
As an alternative, I would recommend you try the Java app as covered in videos 39-42 rather than using Node.js. The Java app does the same thing (loading twitter data into HANA) but as it runs directly in HCP it doesn't encounter the proxy issues.
Thanks,
Philip
Hi Philip,
TOO MUCH THANKS FOR YOUR REPLY!!
I'll try your recommendation and let your know the result at the very first time !
best regards
Joe
Dear Philip,
Sorry to bother again.
but I met a new error when I go through from video 39 to 41
As guided in video 41, when I run my project on server , it turned out to show:
The website declined to show this webpage
Most likely causes:
What you can try:
This error (HTTP 403 Forbidden) means that this program was able to connect to the website, but it does not have permission to view the webpage.
Hi.
This looks like an authorizations issue.
Did you double check to have followed all of the steps in the videos?
Philip
Hi Philip,
Exactly I just followed video 1-11 and got stuck there, then I directly followed video start with 39 and got stuck in video 41.
which video expressed the authorizations issue?
Best Regards
Joe
Hi Joe,
The TwitterJava role referenced in video 41. The Java app may need to be restarted in HCP for the settings to take effect.
Thanks,
Philip
Dear Philip,
Really thank you for your constantly help in SCN, however I still can not handle my problem with load real-time twitters into hana. May I bother you via the skype and give you my control for letting you check how to fix my issue..please..
Deeply thanks
Joe
Hi Philip,
Is it possible to expose via web services the tweets, while the script is running?
for example, what you see in the cmd screen to be displayed in a web browser.
Best,
.CV
Hi Christoforos,
Each tweet is immediately available via web services as soon as it has been loaded into HANA.
Simply use the Tweets entity of the HANA XS Data service (services.xsodata):
Live3HCP/services.xsodata at master · saphanaacademy/Live3HCP · GitHub
Thanks,
Philip
But not from the TA_ table right ?
You have to stop the script, then do the sentiment analysis that will create the TA_ table, and then will be also available, right ?
.CV
The TA_ table (= Text Analytics index) is updated incrementally as and when new tweets are loaded to the Tweets table. No need to stop the script for this. The clustering is re-run on a schedule - every few minutes. So you always get the latest info via web services with no need to stop the script. Philip
What I thought is that you should apply the function each time that you want analyse new tweets, am I wrong ?
No need to apply manually. You simply start the data loading and can use the UI (or any other tool that can access the HANA tables and views) to analyse the results. The tables are auto-refreshed by the system as new tweets are loaded. Philip
Thank you for clarifying this 🙂
I will come back in short with more questions !
Best,
.CV
Hi
I seem to get stuck at this step : video 11 for Load tweets. Here is the error message. Could somebody help me resolve? Thanks in advance.
Hi Sujith,
Sounds like it may be a proxy issue. Are you behind a corporate firewall? If you're unable to get direct access to the internet from this machine consider using the HCP Java app rather than nodeJS for Twitter loading. See videos 39-42.
Thanks,
Philip
Thanks for the quick reply Philip.
Started on that, based on one of your earlier replies here, but stmbled now on this 😥
Does that mean I have to delete the current version of Java and install an older version?
BTW, yes I was behind the SAP corporate firewall. I tried the SET PROXY ... commands and still didnt work.
Hi Sujith, There is a limitation in the nodejs oauth library that the project incorporates in that it doesn't recognize proxies. Best to get direct access to internet (without a proxy) if you can. Thanks, Philip
Hi All
Overcame the previous hurdles, and now on this one 😛 Maybe an issue due to the corporate proxy, can someone help.
The destination connection check works
However when trying out using the Web IDE it gives a gateway timeout
The proxy run screen shows like this at the same time:
What could be the problem? How can I resolve?
Thanks in advance,
Sujith
Hi Sujith,
Have you tried to run this with direct access to the internet i.e. not behind a corporate firewall?
Thanks,
Philip
Yes tried yesterday from home and it did work, Yay! 😎
But why doesnt this work behing the corporate firewall? What do I do to demo it in office?
I am not able to connect to HANA MDC Trial instance using Node JS.
Node JS returns with following error-
app.js file contains following entry-
var hdb = hdb.createClient({
host : 'hana',
port : 30015,
user : 'DEV01',
password : process.argv[2] // from command line
});
Can somebody please guide me on what exactly I have to change in the app.js file?
Hello,
Please note the comments at the top of this blog: Due to recent changes in the features offered in the SAP HANA Cloud Platform Trial Edition, some parts in the scenario below no longer reflect the current situation.
This scenario was created prior to HANA MDC being available in the developer trial so the instructions are no longer current.
Thanks,
Philip