Skip to Content

Screen Shot 2015-03-13 at 14.14.04.png

UPDATE March 15, 2016

Thank You for you interest.  Due to recent changes in the features offered in the SAP HANA Cloud Platform Trial Edition, some parts in the scenario below no longer reflect the current situation.

We are considering to update this playlist in the near future but do not have an estimated timeframe as yet.

Last month, Philip Mugglestone of the SAP HANA Academy recorded a new series of tutorial videos in which he shows how you can build an end-to-end real-time social media analytics application using the SAP HANA Cloud Platform.

The playlist of the series contains 37 videos in total, total duration almost 5 hours. The development project and application is called Live3.

The source data comes at no cost from a social media feed. The services are provided by the SAP HANA Cloud Platform developer edition, a free to use Platform as a Service (PaaS). The developer tools are available for the Windows, Linux and MacOS platforms, at no charge and this concerns two SAP plugins for the Eclipse IDE, the SAP HCP SDK and the open source Google Chrome runtime, Node.js. There are no prerequisites. you can start from scratch.

YouTube Playlist: Live3 – SAP HANA Cloud Platform for Advanced Real-Time Social Media Analytics – YouTube

All content from this project is available on Github https://github.com/saphanaacademy/Live3HCP

Screen Shot 2015-03-13 at 14.28.46.png

Project Description

This enablement offering helps you get started with the SAP HANA Cloud Platform developer edition by walking through a step-by-step project to build a mobile application using real-time data feeds that combines sentiment analysis with predictive analytics whilst exploiting native development features of the SAP HANA Cloud Platform.

The Live3 project allows the CMO to better understand social network sentiment relating to his company, services, or products in order to identify those in a position of influence that publish negative sentiment – so that this segment they can be recruited into a specialized education program. The insight evolves in real-time as new data is posted to social media sites.

Live3 a native SAP HANA application built using the SAP HANA Cloud Platform – leveraging real-time data feeds from social media, data modeling in SAP HANA tools for Eclipse, native SAP HANA text analytics for sentiment analysis as well as native SAP HANA clustering for segmentation, SAP HANA extended application services (XS), OData, and SAP Web IDE to build an HTML5 application using SAPUI5. It can also easily be deployed as a native mobile app.

Whether you are new to SAP HANA Cloud Platform, or just looking to expand your knowledge, this is a great opportunity to gain hands-on experience with the SAP HANA Cloud Platform.

Screen Shot 2015-03-13 at 14.40.31.png

1. Introduction

In the first video Philip discusses the components of the scenario to analyze streaming Twitter data using the SAP HANA Cloud Platform developer edition.

https://youtube.com/watch?v=la9V2hAeuK8

See also Tom’s post: [SAP HANA Academy] Live3: Introduction

2. Register for SAP HANA Cloud Platform Trial

In this video we will register for the SAP HANA Cloud Platform developer edition and create our first SAP HANA instance.

https://youtube.com/watch?v=ld4_V41QVfk

See also Tom’s post: [SAP HANA Academy] Live3: Register for SAP HANA Cloud Platform Trial

3. Install SAP HANA Cloud Platform Tools

In this video after installing the JDK for Java we will download and install SAP HANA Cloud Platform tools including the Web SDK and Eclipse Luna. We will also install SAP HANA Cloud Platform Tools and SAP HANA Tools into the Eclipse environment.

https://youtube.com/watch?v=Hh9ElqQarxU

See also Tom’s post:[SAP HANA Academy] Live3: Install SAP HANA Cloud Platform Tools

4. Add Cloud System

In this video we will add a cloud system from the Eclipse SAP HANA Administration perspective in order to connect to our SAP HANA Cloud Platform developer edition instance.

https://youtube.com/watch?v=mThzHh06iCk

See also Tom’s post: [SAP HANA Academy] Live3: Add Cloud System

5. Identify & Set Application Schema

In this video we will identify and set our application schema in the Eclipse SAPHANA Administration perspective.

https://youtube.com/watch?v=arXF02oLWC0

See also Tom’s post:[SAP HANA Academy] Live3: Identify & Set Application Schema

6. Create Tweets Table

In this video we will create a new SAP HANA database table to contain tweets that we are interested in analyzing further. The data to be loaded will be obtained in real-time via the Twitter API.

https://youtube.com/watch?v=y50lnzHCpP8

See also Tom’s post: [SAP HANA Academy] Live3: Create Tweets Table

7. Setup Twitter

In this video we will setup Twitter as a pre-requisite to using the Twitter API. After reviewing the Twitter API documentation, we will create a Twitter application and generate the consumer and access token keys required to authenticate to the Twitter streaming API.

https://youtube.com/watch?v=YI85ZECYsZo

See also Tom’s blog post:[SAP HANA Academy] Live3: Setup Twitter

8. Setup node.js

In this video we will install node.js.

https://youtube.com/watch?v=gM4SGBxXJeA

See also Tom’s post: [SAP HANA Academy] Live3: Setup node.js

9. node.js Application

In this video we will review the node.js application that creates a local web server which accesses the Twitter streaming API, receives and parses tweets, then loads them into the Tweets table off our SAP HANA database.

https://youtube.com/watch?v=VCWMJoDEmWY

See also Tom’s blog post: [SAP HANA Academy] Live3: node.js Application

10. Setup HCP Tunnel

In this video we will configure and start a tunnel that enables our Node.JS application to communicate directly with our SAP HANA Cloud Platform trial instance.

https://youtube.com/watch?v=pjZiWDjdrzI

See also Tom’s blog post: [SAP HANA Academy] Live3: Setup HCP Tunnel

11. Load Tweets

In this video we will use the Node.JS application to query the Twitter streaming API and load desired tweet information into our SAP HANA instance within the SAP HANA Cloud Platform.

https://youtube.com/watch?v=O_dQzkE8Xcc

See also Tom’s blog post: [SAP HANA Academy] Live3: Load Tweets

12. Text Analysis

In this video we will perform text analysis using the native capabilities provides by SAP HANA in order to identify the sentiment of persons sending tweets – for example strong positive, weak positive, weak negative, or strong negative sentiment.

https://youtube.com/watch?v=Dmo-ekJUe4M

See also Tom’s blog post: [SAP HANA Academy] Live3: Text Analysis

13. Influence & Stance View

In this video we will create a view containing scores for each tweeter based on  influence and stance (attitude). Influence reflects how influential the tweeter is – for example # retweets and replies received by a user compared to retweets and replies sent by the user. Stance reflects whether the user is generally positive or negative based on the sentiment derived during text analysis. The influence and stance scores will be input to the upcoming clustering (or segmentation) step.

https://youtube.com/watch?v=PEt25w9EjGI

14. Explain Clustering

In this video we will learn more about the SAP HANA predictive analysis library (or PAL) and how it can be used to cluster (or segment) tweeters based on their influence and stance scores. We will review the k-means clustering algorithm.

https://youtube.com/watch?v=_9s_hIuiMjk

15. Perform Clustering

In this video we will review and run the SQL Script necessary to setup and execute k-means clustering analysis using the SAP HANA predictive analysis library. We will also create some views to be accessed subsequently via web services.

https://youtube.com/watch?v=xEGX64AUDiM

16. SAP HANA Web-based Development Workbench

In this video we will introduce the SAP HANA Web-based Development Workbench and create our live3 web services project.

https://youtube.com/watch?v=cNmtNT-1EnQ

17. Web Services – Authorizations

In the this video we will configure authorizations for web services. Application privileges and user roles will be defined.

https://youtube.com/watch?v=qs061nzBdyU

18. Web Services – Setup OData

In this video we will introduce what OData is and set up an OData service as part of our live3 web services project.

https://youtube.com/watch?v=KjsYgAmUwHY

19. Web Services – Using OData

In this video we will explore the OData web services created previously and make some example queries.

https://youtube.com/watch?v=0MQgsMsSZjc

20. Web Services – Setup XSJS

In this video, we will add server-side scripting capabilities to our web services project. This will allow us to configure actions to refresh the clustering and “reset” the database.

https://youtube.com/watch?v=ZdKk_VXCwUw

21. Web Services – Debugging XSJS

In this video we will see how server-side scripting applications can easily be debugged using the SAP HANA Web-based Development Workbench.

https://youtube.com/watch?v=24GLegisPqY

22. Web Services – Authentication

In this video we will understand why a “proxy” authentication server is needed to access your SAP HANA Cloud Platform web services from a SAP HANA Cloud HTML5 application.

https://youtube.com/watch?v=Kaun6PPupzA

23. Web Services – Authentication Setup Proxy

In this video we will setup the “proxy” authentication server.

https://youtube.com/watch?v=fhtbO532DiI

24. Setup SAP HANA Cloud Connector

In this video we will install and configure the SAP HANA Cloud Connector.

https://youtube.com/watch?v=jqGfXL9Vz00

25. Setup SAP HANA Cloud Connector – Resources

In this video we will define SAP HANA Cloud Connector resources in order to make back-end systems available to the SAP HANA Cloud Platform.

https://youtube.com/watch?v=_0gHqaf2Rvw

26. Setup SAP HANA Cloud Platform – Destinations

In this video we will define SAP HANA Cloud Platform “destinations” to reference back-end resources previously configured using the SAP HANA Cloud Connector.

https://youtube.com/watch?v=1_E7MOyCJ4k

27. Setup SAP Web IDE

In this video we will setup SAP Web IDE so that we can more easily develop our HTML5 application based on SAPUI5.

https://youtube.com/watch?v=jWM1i82qr5o

28. SAP Web IDE – Templates

In this video we will use SAP Web IDE to generate an SAPUI5 application driven by the OData services we created previously with SAP HANA XS. We will introduce and use the SAP Web IDE “Master Master Detail” template.

https://youtube.com/watch?v=myJhpRiLs10

29. SAP Web IDE – Deploy HTML5

In this video we will use SAP Web IDE to deploy our SAPUI5 project as a fully fledged SAP HANA Cloud Platform HTML5 application.

https://youtube.com/watch?v=0dhffxH0EFQ

30. SAP Web IDE – Code Review

In this video we will review the structure and code that was generated by the “Master Master Detail” template of SAP Web IDE.

https://youtube.com/watch?v=NSQJhQR8JU8

31. SAP Web IDE – Customize Master – List

In this video we will customize the code that was generated by the “Master Master Detail” template of SAP Web IDE. We will add additional attributes to the list in the master view.

https://youtube.com/watch?v=X98CIrgdmf8

32. SAP Web IDE – Customize Master – Toolbar

In this video we will customize the code that was generated by the “Master Master Detail” template of SAP Web IDE. We will add a Pull To Refresh control to the master view and an icon toolbar to the footer.

https://youtube.com/watch?v=-HVKAbwANY0

33. SAP Web IDE – Customize Master – Methods

In this video we will customize the code that was generatedby the “Master Master Detail” template of SAP Web IDE. We will modify the search method and add new methods for the toolbar icons.

https://youtube.com/watch?v=RNYK1HvnRW0

34. SAP Web IDE – Customize Master – Test

In this video we will customize the code that was generated by the “Master Master Detail” template of SAP Web IDE. We will configure and test the toolbar icons to perform clustering, reset, and to start/stop real-time loading of data via the Twitter API.

https://youtube.com/watch?v=sGVm5ENw7Hs

35. SAP Web IDE – Customize Master 2 & Detail

In this video we will customize the code that was generated by the “Master Master Detail” template of SAP Web IDE. We will add additional attributes to the list in the master2 view and a list of tweets to the detail view.

https://youtube.com/watch?v=AEyWk-iQVIc

36. SAP Web IDE – Customize Master 2 Controller

In this video, we will customize the code that was generated by the “Master Master Detail” template of SAP Web IDE. We will fix an issue in the master2 controller that causes the details view to be refreshed unnecessarily.

https://youtube.com/watch?v=EaRY9hTWiuM

37. SAP Web IDE – Commit Push HTML5

In this video we will use SAP Web IDE to deploy our SAPUI5 project as a fully fledged SAP HANA Cloud Platform HTML5 application. We will stage, commit and push our changes and activate a new app version.

https://youtube.com/watch?v=jZUY-cGJuuw

38. Test With Mobile Device

In this video we will test the completed application on an iPhone.

https://youtube.com/watch?v=i97PD-_e3rs

39. Java – Create Project

In this video we cover an alternative approach to loading Twitter data into SAP HANA data via Node.js. We will create a Java application. After setting up Eclipse for Java development with SAP HANA Cloud Platform, we will create the Java project then download the Java source code from Github and perform some initial configuration.

https://youtube.com/watch?v=d3ixBc6bW9s

40. Java – Souce Code Walkthrough

In this video, Mac Holden walks us through the source code of the Java project.

https://youtube.com/watch?v=z_-Ru-ygKes

41. Java – Deploy to HCP and Test

In this video, we configure appropriate HCP roles and data bindings, test the application, and load new tweets into SAP HANA.

https://youtube.com/watch?v=hIev1FeqR-k

42. Java – Update HTML5 App, Deploy and Test

In this video, we will create a HCP destination for our Java application and make the necessary source code changes to the HTML5 app with SAP Web IDE, then update the deployed version and verify that the end to end scenario

https://youtube.com/watch?v=Cd_uft_282I

To report this post you need to login first.

38 Comments

You must be Logged on to comment or reply to a post.

  1. Christoforos Verras

    Hello Philip and thank you for these great series !

    Unfortunately there is a problem on quering the tweets.

    Although i enter “/do/start?track=%23SAPHANA%20” translating to #SAPHANA(space)

    i get tweets that have nothing to do with this.

    Especially from Greece. (where i live)

    Is there any chance to me synced with my twitter account ?

    What am i doing wrong?

    Best regards,

    Christoforos Verras

    <edit>:

    after lots of “debugging” i am now sure that it tracks also the tweets from the users i follow.

    A solution might be to create a new account with no followers at all..

    <edit>:

    (0) 
  2. Rahul Mehta

    Hi Philip,

    From past couple of weeks I have been extensively working on HCP and building Iot related scenarios. I came across one for your scenario of reading the tweets, processing it in XS engine and displaying it in nice HTML5 app.

    First of all I must say its an amazing video showcasing the end to end scenario and really helpful for the people working on HCP.

    In the set of those 37 videos, there is a video where you explain the usage of HANA Cloud connector and why is it necessary to use CC.

    I want to understand couple of things here where I need your confirmation and guidance.

    1) I have an Odata provider which comes from on premise ECC system.

    2) I have an HTML5 app deployed on HCP.

    3) I need to now create a destination in HCP to backend ECC system. For this I mandatorily need to use Cloud connector right ?

    4) In your example you have used a proxy to connect html5 and native hana xs using CC. But this is only applicable for trial instance right. In the actual environment we don’t need to do that and HTML5 can directly connect to HANA XS via destination configuration? 

    Thanks in advance.

    (0) 
    1. Christoforos Verras

      Hi Rahul ,

      I have a question about the Twitter application.

      Do you only track for the word you want, or did you ever try to search for previously published tweets ?

      What I am trying to do is to use the search API instead of the streaming API.

      Any help would be appreciated  !

      Best,

      Christoforos Verras

      (0) 
    2. Philip MUGGLESTONE

      Hi Rahul,

      Thanks for the nice feedback.

      CC is needed when you have a back-end system that is not visible on the public internet i.e. behind some kind of firewall. CC runs on a machine behind the firewall and creates a “private tunnel” between that machine and your HCP account – this then means that HCP destinations can reach the back end system without having to make the back end system publicly visible on the internet. Make sense?

      So if you have a back-end system that’s behind a firewall, then yes use CC to enable HCP destinations to reach it. If however your back-end system is already visible on the public internet no need for CC – just point the HCP destinations to it directly.

      Yes, the use of CC to connect to HCP HANA XS only applies to the free developer trial of HCP. In HCP productive systems, HCP destinations can connect & authenticate directly.

      Thanks,

      Philip

      (0) 
      1. Rahul Mehta

        Thanks a lot Philip for clarifying this. I am trying this now with one of our ECC systems behind the firewall. Will get back in case of any issues

        Regards

        Rahul

        (0) 
  3. Rahul Mehta

    Hi Philip

    I have a question regarding setting up the HANA instance in HCP. We have currently set up a default instance. Can we set up a different HANA instance here. Why this question because we have Operational Process Intelligence running on a HANA box and through HANA XS application in HCP I want to push this data to a different HANA box where OPI is installed. Is it possible to do that?

    Regards

    Rahul

    (0) 
      1. Rahul Mehta

        Hi Christoforos

        Actually I was thinking to delete the default instance and create another one pointing to my HANA instance where OPInt is installed. What do you think? Will that work?

        Regards

        Rahul

        (0) 
        1. Philip MUGGLESTONE

          Hi Rahul,

          HCP only allows a single HANA instance in the developer trial. If you delete the one you currently have you can only then create a new empty one. The HCP starter edition in the production landscape allows multiple instances so might be better suited to your needs? Also, in the production landscape you won’t need the hana-auth-proxy (see video 23) to connect the HTML5 app to the HANA instance.

          Thanks,

          Philip

          (0) 
  4. Joe Zhou

    hi  expert,

    I’m really got crazy about this step 11 “load tweets”.  this is the error log:

    Error: getaddrinfo ENOTFOUND userstream.twitter.com userstream.twitter.com:443

        at errnoException (dns.js:26:10)

        at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:77:26)

    everything is ok util I enter the http://localhost:8888/do/start?track=ausopen  the page shows “start” while the cmd shows the error

    PlZ help me !!!!!!!!

    (0) 
  5. Joe Zhou

    hi  expert,

    I’m really got crazy about this step 11 “load tweets”.  this is the error log:

    Error: getaddrinfo ENOTFOUND userstream.twitter.com userstream.twitter.com:443

        at errnoException (dns.js:26:10)

        at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:77:26)

    everything is ok util I enter the http://localhost:8888/do/start?track=ausopen  the page shows “start” while the cmd shows the error

    PlZ help me !!!!!!!!

    (0) 
    1. Philip MUGGLESTONE

      Hi Joe,

      This message sometimes appears when your node.js application is behind a firewall so it cannot reach the Twitter API. In that case it’s necessary to declare the proxy server.

      As an alternative, I would recommend you try the Java app as covered in videos 39-42 rather than using Node.js. The Java app does the same thing (loading twitter data into HANA) but as it runs directly in HCP it doesn’t encounter the proxy issues.

      Thanks,

      Philip

      (0) 
      1. Joe Zhou

        Hi Philip,

        TOO MUCH THANKS FOR YOUR REPLY!!

        I’ll try your recommendation and let your know the result at the very first time !

        best regards

        Joe

        (0) 
      2. Joe Zhou

        Dear Philip,

        Sorry to bother again.

        but I met a new error when I go through from video 39 to 41

        As guided in video 41, when I run my project on server , it turned out to show:

        The website declined to show this webpage

        HTTP 403  
            

        Most likely causes:

        • This website requires you to log in.
            

        What you can try:

            
           Go back to the previous page.
            
        More information   More information  

        This error (HTTP 403 Forbidden) means that this program was able to connect to the website, but it does not have permission to view the webpage.

        (0) 
          1. Joe Zhou

            Hi Philip,

            Exactly I just followed video 1-11 and got stuck there, then I directly followed video start with 39 and got stuck in video 41.

            which video expressed the authorizations issue?

            Best Regards

            Joe

            (0) 
              1. Joe Zhou

                Dear Philip,

                Really thank you for your constantly help in SCN, however I still can not handle my problem with load real-time twitters into hana. May I bother you via the skype and give you my control for letting you check how to fix my issue..please..

                Deeply thanks

                Joe

                (0) 
      1. Christoforos Verras

        But not from the TA_ table right ?

        You have to stop the script, then do the sentiment analysis that will create the TA_ table, and then will be also available, right ?

        .CV

        (0) 
        1. Philip MUGGLESTONE

          The TA_ table (= Text Analytics index) is updated incrementally as and when new tweets are loaded to the Tweets table. No need to stop the script for this. The clustering is re-run on a schedule – every few minutes. So you always get the latest info via web services with no need to stop the script. Philip

          (0) 
            1. Philip MUGGLESTONE

              No need to apply manually. You simply start the data loading and can use the UI (or any other tool that can access the HANA tables and views) to analyse the results. The tables are auto-refreshed by the system as new tweets are loaded. Philip

              (0) 
    1. Philip MUGGLESTONE

      Hi Sujith,

      Sounds like it may be a proxy issue. Are you behind a corporate firewall? If you’re unable to get direct access to the internet from this machine consider using the HCP Java app rather than nodeJS for Twitter loading. See videos 39-42.

      Thanks,

      Philip

      (0) 
      1. Sujith Philipose

        Thanks for the quick reply Philip.

        Started on that, based on one of your earlier replies here, but stmbled now on this 😥

        /wp-content/uploads/2016/03/publish_error_902822.png

        Does that mean I have to delete the current version of Java and install an older version?

        (0) 
        1. Philip MUGGLESTONE

          Hi Sujith, There is a limitation in the nodejs oauth library that the project incorporates in that it doesn’t recognize proxies. Best to get direct access to internet (without a proxy) if you can. Thanks, Philip

          (0) 
  6. Sujith Philipose

    Hi All

    Overcame the previous hurdles, and now on this one 😛 Maybe an issue due to the corporate proxy, can someone help.

    The destination connection check works

    Dest_connect.png

    However when trying out using the Web IDE it gives a gateway timeout

    /wp-content/uploads/2016/03/gateway_error_908305.png

    The proxy run screen shows like this at the same time:

    /wp-content/uploads/2016/03/cmd_error_908318.png

    What could be the problem? How can I resolve?

    Thanks in advance,

    Sujith

    (0) 
      1. Sujith Philipose

        Yes tried yesterday from home and it did work, Yay!  😎

        But why doesnt this work behing the corporate firewall? What do I do to demo it in office?

        (0) 
  7. TAPAS SANTRA

    I am not able to connect to HANA MDC Trial instance using Node JS.

    Node JS returns with following error-

    Node JS.jpg

    app.js file contains following entry-

    var hdb = hdb.createClient({

      host     : ‘hana’,

      port     : 30015,

      user     : ‘DEV01’,

      password : process.argv[2] // from command line

    });

    Can somebody please guide me on what exactly I have to change in the app.js file?

    (0) 
    1. Philip MUGGLESTONE

      Hello,

      Please note the comments at the top of this blog: Due to recent changes in the features offered in the SAP HANA Cloud Platform Trial Edition, some parts in the scenario below no longer reflect the current situation.

      This scenario was created prior to HANA MDC being available in the developer trial so the instructions are no longer current.

      Thanks,

      Philip

      (0) 

Leave a Reply