HANA smart data streaming is included as an installed component in the HANA Developer Edition for both Amazon Web Service (“AWS”) and Microsoft Azure (“Azure”) hosts. The HANA Developer Edition is a single node HANA system with multiple HANA components pre-installed to help new users get up and running quickly and let the user focus on learning the capabilities of HANA and it’s options without getting bogged down in installation tasks.

In order to start using smart data streaming, there are a few steps that you need to complete.

Create a HANA Developer Edition Instance

Detailed instructions for creating a virtual machine instance of the HANA Developer Edition can be found here: SAP HANA Developer Edition in the Cloud | Tutorial | SAP

You will require an existing account for eithere Amazon Web Services or Microsoft Azure as your HANA Developer Edition instance will be hosted on one of those cloud infrastructure providers.

Install Eclipse and the HANA Plug-Ins

The standard development and administration environment for working with a HANA system is HANA Studio. This is an Eclipse based environment with multiple plug-ins specfic to working with HANA. You will need to install Eclipse on the local PC or laptop that you will be using to connect to HANA. Detailed instructions on installing Eclipse and the core HANA plug-ins are available at: SAP Development Tools for Eclipse

Install the Smart Data Streaming Plug-Ins

HANA smart data streaming requires an additional plug-in that provides both the SAP HANA Streaming Development and SAP HANA Streaming Run-Test perspectives. This plug-in is not currently available in the tools.hana.ondemand.com repository, but rather is included as a .sar file on the HANA Dev Edition instance. The plug-in installation file can be found in the /hana/shared/HDB/streaming directory.

Detailed instructions on installing the plugin can be found in this document here on the Smart Data Streaming Developer Center: Installing the Smart data streaming plugin for HANA Studio Within those instructions, you can skip over the Pre-requisites and Download sections and go straight to the Install section. In step 2 of the Install section where you are instructed to start HANA Studio, you will instead be starting the Eclipse system that you installed earlier.


Configure Your Hosts File

Ideally smart data streaming hosts are added to a HANA system using a fully qualified domain name (“FQDN”). However that isn’t always possible. In the case of the HANA Developer Edition, the HANA host is created with a virtual host name of “vhcalhdbdb”. This is done to facilitate the cloning or publishing process used to create multiple instances of the HANA Dev Edition VM from a single template. Your specific instance of the HANA Dev Edition will run in a VM on either AWS or Azure that is unique to you.

Since your HANA instance will be using the virtual host name of “vhcalhdbdb”, you will need to create an entry in the network “etc\hosts” file on the PC or laptop where you are running Eclipse with the HANA plug-ins. This entry will be used to assign the IP address of your VM to the aliases “vhcalhdbdb”.

On Windows, the hosts file is typically found in the C:\Windows\System32\drivers\etc directory. The entry that you need to create will look like this:

# HANA Dev Edition

52.23.155.102 vhcalhdbdb vhcalhdbdb.wdf.sap.corp

Note that you do need “Run as Administrator” permissions in order to edit the hosts file on Windows.

Next Steps

Once you have completed the above configuration steps, you will be ready to start using HANA smart data streaming. The  Using the Freezer Monitoring Tutorial for SAP HANA Smart Data Streaming is a great starting point for new users that will walk you through all of the steps from creating a streaming project and connecting to the streaming server to compiling, running and testing the project. Many of our other tutorials available here on the SCN have been written to extend your learning based on using the Freezer Monitoring tutorial as a starting point.

Note on Project Configuration when working with Cloud Servers

Each individual streaming requires 3 TCP ports to be available in order to run. By default these ports are allocated dynamically when the project is started. This behavior works well in on-premise environments where all ports are normally open within the local internal network. In contrast, when working with external cloud based servers, a standard best practice is to only open the minimum set of ports required for the application.

The HANA Dev Edition instance is configured with ports 65532, 65533 and 65534 already open for use by a streaming project. This blog post provides instructions on how to edit the default project configuration to have your streaming project use specific ports when starting up, rather than the dynamically allocating ports which is the default behaviour. Specifying the Command Port to be used by a Smart Data Streaming Project

If you wish to run more than one streaming project at time, then you will need to open 3 additional ports per project and configure each project to use a unique set of ports.

To report this post you need to login first.

32 Comments

You must be Logged on to comment or reply to a post.

  1. Svenja Brunstein

    Hi Robert,

    thanks for this great overview.

    I configured the hosts file as you described above and opened the ports in our firewall. But when starting a streaming project from HANA studio, this message is logged into the console:

    Failed call to:https://vhcalhdbdb:65535/RPC2 (Failed to read server’s response: Unable to tunnel through proxy. Proxy returns “HTTP/1.0 404 Not Found”)

    Any idea on what might cause this?

    Thanks,

    Svenja

    (0) 
    1. Robert Waywell Post author

      In HANA Studio, under Window->Preferences->General->Network Connections what is the Active Provider set to?

      HANA Studio - Network Preferences.png

      Mine is set to “Native”, you may need to set yours to “Direct” to skip the Proxy.

      (0) 
        1. Robert Waywell Post author

          That’s good. The open ports for the project to run on are actually 65532,65533 and 65534. We had to move off of 65535 due to a port conflict. So in your project configuration if you switch port 65535 to 65532 then you should be good.

          I’ll update the linked Command Port configuration document to be consistent.

          (0) 
  2. Xiaochuan Wang

    Hi, I install Smart data streaming plug in in my HANA studio(windows),

    but I cannot find any input/output adapter in my ccl diagram.

    Do you have any idea about this issue ?

    NullAdapter.PNG

    (0) 
    1. Robert Waywell Post author

      Hi Xiaochuan,

      In order to succesfully install the plugins on Windows, you need to start HANA Studio with “Run as Administrator” privileges. The symptom of not seeing the adapters show up in the palette usually indicates that HANA Studio was not started with “Run as Administrator” privileges when you installed the plugin.

      You should be able to resolve the issue by:

      1) Restart HANA Studio with “Run as Administrator” privileges

      2) Uninstall the SAP HANA smart data streaming Plugin

      3) You will need to restart HANA Studio for the uninstall to complete. Rather than letting Studio automatically restart, you want to manually exist Studio and then manually restart it so that you can start Studio with “Run as Administrator” privileges.

      4) You can now re-install the SAP HANA smart data streaming Plugin and the elements should properly show up in the palette.

      (0) 
      1. Xiaochuan Wang

        Hi Robert:

        I’m using SDS to process data stream via continuously querying external database.

        And I referred to below guide:

        http://help.sap.com/Download/Multimedia/zip-hana_options_sds/streaming_adapters_guide.pdf

        In this document, variable ?ESP_MSDATE_VALUE is required to set in my SQL.

        I’m confused with it, is the variable’s name fixed to ESP_MSDATE_VALUE?

        And when I use it to query, error occurs, I checked log file project.trace as follows:

        [23934]{-1}[-1/-1] 2016-03-07 16:46:42.187000 e Streaming UNKNOWN(-0001) : [SP-3-100005] (1.220) sp(23866) Error executing SQL statement SELECT  COLUMN1, COLUMN2, COLUMN3 FROM “WXC_AUTO” WHERE COLUMN3 > ?ESP_MSDATE_VALUE. java.sql.SQLException: Missing IN or OUT parameter at index:: 1

        (0) 
        1. Remi ASTIER

          Hello Xiaochuan,

          Did you specify adapter property msdateSequenceColumn to COLUMN3 ?

          Could you switch to text authoring and paste the code for the adapter ?

          Remi.

          (0) 
          1. Xiaochuan Wang

            Hi Remi:   Here is my project CCL:

            CREATE INPUT STREAM InputStream1 SCHEMA (   

              C1 bigint ,    

              C2 string ,   

              C3 msdate

            ) ;

            ATTACH INPUT ADAPTER Generic_DB_Input1 TYPE db_in TO InputStream1 PROPERTIES   

              service = ‘ds_oracle’ ,    

              query = ‘SELECT  COLUMN1, COLUMN2, COLUMN3 FROM \”WXC_AUTO\” WHERE   COLUMN3 > ?ESP_MSDATE_VALUE’ ,     table = ‘\”\”.\”WXC_AUTO\”‘ ,    

              pollperiod = 10 ,   

              seconddateFormat = ‘%Y-%m-%d %H:%M:%S’ ,    

              msdateFormat = ‘%Y-%m-%d %H:%M:%S’ ,    

              permutation = ” ,   

              msdateSequenceColumn = ‘COLUMN3’ ,   

              recordCounterColumn = ” ,   

              msdateSequenceInitValue = ‘2015-01-01 16:07:10’ ,   

              recordCounterInitValue = ‘0’ ,    

              blockSize = 5000 ,    

              reconnectAttemptDelayMSec = 1000 ,   

              maxReconnectAttempts = 1 ;

            (0) 
            1. Remi ASTIER

              Ok.

              When written manually, the code is easier to read. Could you try this code :

              ATTACH INPUT ADAPTER Generic_DB_Input1 TYPE db_in TO InputStream1

              PROPERTIES     service = ‘ds_oracle’ ,

                   query = ‘SELECT  COLUMN1, COLUMN2, COLUMN3 FROM “WXC_AUTO” WHERE COLUMN3 > ?ESP_MSDATE_VALUE ORDER BY COLUMN3’ ,

                   pollperiod = 10,

                   msdateSequenceColumn = ‘COLUMN3’,

                   msdateSequenceInitValue = ‘2015-01-01 16:07:10’;

              (0) 
              1. Xiaochuan Wang

                Hi Remi:

                     Unfortunately, it still cannot work with your code. And same error occurs which can be

                checked in trace file.

                    Actually I’m puzzled with this msdateSequenceColumn, does it mean the database table column name or Stream/Window schema column name ?

                oracle.PNG

                (0) 
                1. Remi ASTIER

                  Try this, it should create the query automatically :

                  ATTACH INPUT ADAPTER Generic_DB_Input1 TYPE db_in TO InputStream1

                  PROPERTIES     service = ‘ds_oracle’ ,

                       table = ‘WXC_AUTO’,

                       pollperiod = 10,

                       msdateSequenceColumn = ‘COLUMN3’,

                       msdateSequenceInitValue = ‘2015-01-01 16:07:10’;

                  (0) 
                  1. Xiaochuan Wang

                    It is not OK. The error is:

                    Error binding parameters to SQL statement select * from WXC_AUTO. java.sql.SQLException: Invalid column index.

                    (0) 
                  2. Xiaochuan Wang

                    I guess the point is how to pass last query msdate to SQL clause, in which a placeholder

                    ? locates.

                    I didn’t see any explanation about it.

                    (0) 
                  3. Xiaochuan Wang

                    I think it is a mistake in the SDS guide.

                    I checked the file “./STREAMING-1_0/bin/i18n/server/esp-internal-adapters-errors.properties”, and found

                    捕获.PNG

                    So it should be ?ESP_TIMESTAMP_VALUE, but not ?ESP_MSDATE_VALUE.

                    (0) 
                    1. Robert Waywell Post author

                      Hi Xiauchuan,

                      Did you try changing the parameter name in the project to ESP_TIMESTAMP_VALUE? If so, did that resolve your issue?

                      Thanks

                      (0) 
  3. Peter Hrebik

    Hi Robert,

    I am having problem with setting up the Data Services in Eclipse:

    Screen Shot 2016-04-07 at 13.33.29.png

    I created the service as follows:

    Screen Shot 2016-04-07 at 13.34.13.png

    Screen Shot 2016-04-07 at 13.34.58.png

    What am I missing here? I am running SAP HANA Dev edition in AWS. Any advice much appreciated!

    (0) 
    1. Robert Waywell Post author

      Hi Peter,

      What you should be seeing in the Add Service popup menu in your 2nd screen shot is an option to add a HANA Service. One possible reason why you aren’t seeing the Add HANA Service option would be that you got an older version of the SDS plugin but if you got the plugin from the Dev Edition VM that is pretty unlikely.

      A 2nd possibility is that you may have installed the plugin without first starting Eclipse with Run as Administrator permissions on Windows. In that case you would need to restart Eclipse with Run as Administrator permissions, uninstall the SDS plugin,  restart Eclipse again Run as Administrator permissions to complete the uninstall and then reinstall the plugin.

      Can you confirm if you also have the HANA Administration Console perspective installed in your Eclipse environment? I don’t know of any dependencies on other perspectives, but the Admin Console is one that would be installed by default in HANA Studio and which you may not have installed yet in your base Eclipse environment.

      Thanks

      Rob

      (0) 
      1. Peter Hrebik

        Hi Robert,

        thanks for reply. I am still struggling with this.

        I am running Eclipse on Mac. I found the Hana studio plugins on Hana Dev Edition instance in the folder /hana/shared/HDB/hdbstudio_update. Can you confirm, are these the plugins you have been referring to? I was unable to install them on Mac. Eclipse (4.4 Luna) simply won’t “pick them up”. I was able to install them on Windows machine (Eclipse 4.5 Mars):

        Screen Shot 2016-04-08 at 12.11.52.png

        I did of course run the Eclipse as Administrator.

        After plugins installation, following perspectives have been added:

        Screen Shot 2016-04-08 at 12.57.07.png

        Streaming development perspectives are not present.

        I tried to install all the HANA plugins from https://tools.hana.ondemand.com/mars:

        Screen Shot 2016-04-08 at 12.56.44.png

        Streaming perspectives are still not present.

        Any idea what I am missing here? I know I can get the SDS plugins from SAP Service Marketplace but that’s what I did in Eclipse on Mac. The result was missing “Add HANA service” menu item!

        (0) 
          1. Peter Hrebik

            Hi Robert,

            sorry for late reply. I’ve been away for some time. Please see attached screenshot. Thanks in advance for looking into this.Screen Shot 2016-07-05 at 12.13.14.png

            (0) 
            1. Robert Waywell Post author

              To make sure I’m on the same page – you still aren’t seeing the perspectives listed in the Open Perspective dialogue, correct?

              HANA Studio - Open Perspective.png

              (0) 
                1. Robert Waywell Post author

                  When you select the SAP HANA Streaming Development perspective, does it open up? If so, then I think you have it working now.

                  (0) 
                  1. Peter Hrebik

                    Yes I get the SAP Hana Streaming Development perspective open ok, however I still cannot setup the Data Services (see my initial comment). I am not able to see the “Add HANA” service menu item on the right-click menu. I cannot see it on Windows nor Mac. Any idea?

                    (0) 
                  2. Peter Hrebik

                    Hi Robert,

                    Ok, so I did update my Smart data streaming plugin to 1.10.12.v201605261044. I am still not able to see the “Add Hana service” menu item. I did terminate the SAP Hana Dev edition SPS10 and created new SAP Hana Dev SPS11 instance.

                    1. I successfully added the system in the SAP HANA Administration Console
                    2. I can connect to the system fine
                    3. In the SAP Hana Streaming Run-Test perspective I’ve added new server connection (esps://<MY HOST>:30026):

                      Host name: the IP of the SAP Hana Dev Instance

                    Port: 33026

                    Protocol: Native

                    SSL: enabled

                    Authentication: User/Password

                                4. In the SAP Hana Streaming Development perspective I am not able to see the “Add Hana service” menu item

                    How can I connect to the streaming server? I tried to created ODBC service with the ODBC DSN: hanaservice with no success.

                    Please help. I am having exactly same issue on Mac and Windows. The SAP Hana Dev instance comes with the streaming server already pre-installed, also the streaming server shows as running in the management console. Any idea what I am missing here?

                    (0) 
                    1. Robert Waywell Post author

                      At this point it isn’t obvious what is wrong. The fact that you can get as far as the pop up menu to add a data service shows that the plug in is installed and you are connected to the SDS server. We’ve already checked that you have the SP10 plugin and since you are connecting to the SP11 Dev Edition server we know that you are connecting to an SDS instance that provides the HANA Service feature.

                      Since you have moved from the SP10 Dev Edition to the SP11 Dev Edition, you can try upgrading the SDS plugin to SP11. That would be recommended regardless as there are SP11 features that aren’t supported by the SP10 SDS Studio plug in such as the machine learning functions.

                      I will try to find time to test this with a plain Eclipse install to see if I can reproduce it.

                      (0) 
                      1. Peter Hrebik

                        Hi Robert,

                        just to let you know I managed to solve my issue by uninstalling the Eclipse and instead of installing HANA SDS plugins into existing Eclipse IDE, I downloaded SAP Hana Studio 2 from SAP Marketplace. After installing SAP hdstudio 2 all works fine!

                        Thanks

                        Peter

                        (0) 
  4. Biswanath Sahu

    I am looking for the Eclipse plugin for SDS.

    How can I download the plugin installation from /hana/shared/HDB/streaming of the Cloud developer edition to my local machine.

    Will really appreciate your help!!

    Thanks and Best Regards,

    Biswa

    (0) 

Leave a Reply