What’s New In SPS11 for HANA Smart Data Streaming
Highlights of the HANA smart data streaming (SDS) SPS11 release:
- Introduction of Machine Learning Algorithms for stream processing
- SPS11 introduces 2 new machine learning algorithms in SDS. These are native CCL functions that can be utilized directly within streaming projects. These have been implemented by the PAL team and the chosen algorithms are specifically designed to operate against live data streams in real-time.
- A Getting Started video is now available in the Smart Data Streaming playlist of the HANA Academy YouTube channel: SAP HANA Academy – Smart Data Streaming: Getting Started with Machine Learning [SPS 11] – YouTube
- Adaptive Hoeffding Tree – An incremental decision tree algorithm which uses limited samples to choose the best tree node splitting attribute
- HANA Academy videos showing how to create the model and the streaming project are available at:
- HANA Academy videos showing how to use the model for scoring are available at:
- Denstream – An incremental clustering algorithm which uses the concept of micro-clusters to summarize clusters of arbitrary shapes and an elaborated pruning technique to detect outliers.
- HANA Academy videos showing how to create the Denstream model and build a streaming project using the model are available at:
- SPS11 introduces 2 new machine learning algorithms in SDS. These are native CCL functions that can be utilized directly within streaming projects. These have been implemented by the PAL team and the chosen algorithms are specifically designed to operate against live data streams in real-time.
- Support for HANA System Replication
- SPS11 introduces SDS support for HANA System Replication
- SDS nodes will be created in the replica system
- Deployed SDS projects will be copied to the replica
- Streaming projects will require a manual restart to allow coordination with outside data sources
- State information for running projects including log files for log backed windows are not replicated in SPS11. Support for this functionality is on the roadmap for a future release
- Adapter Enhancements
- Simplified Configuration of the SAP HANA Output Adapter
- Older options that explicitly set the commit interval, number of events to be committed in a single batch and the number of worker threads to be used have been replaced by new parameters that set upper limits for each of these values that the adapter will then work within
- The total number of parameters that can be set for the HANA Output Adapter has been reduced. The adapter now automatically adjusts the insert array size, the max queue size and the maximum length of time to buffer data before outputting it to HANA.
- Custom Header Protection Support for the HTTP Client Output Adapter
- This is an important new feature for streaming projects that integrate to an XS application. XS application enforce the use of a Cross-Site Request Forgery (CSRF) token.
- With this feature, the HTTP Client Output Adapter can now automatically request and use a CSRF token
- New Custom Adapter File Directory
- SDS supports the development, integration and use of custom adapters to provide integration options to data sources not already supported by our ‘out-of-the-box’ adapters
- Custom adapters built using the Adapter Toolkit can be integrated directly into the palette in the SAP HANA Streaming Development perspective in HANA Studio
- To provide better organization of custom adapters, the custom adapter files are now stored in the STREAMING_CUSTOM_ADAPTERS_HOME directory
- Simplified Configuration of the SAP HANA Output Adapter
- Streaming Web Service (SWS) Enhancements
- In SPS 10, the SWS is configured, started and stopped as a separate process. While it was often run on the same host as the SDS server node, it still had to be manually started and stopped.
- With SPS 11, the SWS is now managed as part of the SDS cluster and can be configured to automatically start when the SDS cluster starts
- If you are upgrading from SPS10, then there are some one time manual migration steps required to migrate your current SWS settings from the old configuration file into HANA Cockpit
- Data Service Enhancements
- This is a security enhancement that provides greater control over who has access to specific data services. Prior to SPS11, any data service was visible across all workspaces on the SDS server. With SPS11 you can now choose to limit the data service availability to a single workspace, or continue to make it available across all workspaces on the server.
Incremental Enhancements Based on User Feedback
- Added multi-threading support for custom row based formatters used in building custom adapters
- Support for additional dynamic attributes for file/hadoop adapters
- New methods have been added to the C, .NET and Java SDKs to process formatting errors with fixed decimal values
Hi Robert,
Thanks for the very informative post. There are lots of new features that I find most intersting and likely very useful.
We currently have SAP ESP installed in our organisation. Is there a planned release for ESP that will contain a number of the same features? Or is the SDS product becoming increasingly different from ESP?
Thanks,
Jason.
Hi Jason,
Yes we are seeing some differentiation between SDS and ESP which reflects a greater investment in SDS as part of the integrated HANA platform. Another relevant factor is that the HANA platform is only supported on Linux while ESP is supported on additional platforms. This means that releasing the same new features on ESP requires additional additional development and QA effort over releasing on SDS only. That said, several enhancements are still shared between SDS and ESP.
Are there particular new features that you are seeing in SDS that you would like to see in ESP?
Understanding that you are using ESP today, do you also use HANA and is there the opportunity to move to SDS?
Thanks
Rob
Hi Robert,
I apologize for the invasion, but I'm also very interested to understand the differences between ESP and SDS roadmaps.
Our company uses BW on HANA and S/4HANA will be. At the moment we used old version of ESP (SP04) and I want to upgrade it. Currently we are testing the capabilities of SP10 and I can already see a forthcoming transition. As for the SP10 if I understood correctly ESP and SDS is very similar but what's about the future releases?
Regarding opportunity to move to SDS this question may appear on the agenda.
Best Regards,
Michael
Hi Michael,
While many enhancements and new features will continue to be shared between ESP and SDS, it is likely that there will also be some new features which are only available in SDS. In some cases, such as support for HANA System Replication the new feature is only relevant to SDS and isn't relevant to ESP. In other cases where we are enhancing existing shared components such as the HANA Output adapter or the HTTP Client Output adapter in SPS11, the enhancements will continue to benefit both product packages. Then there will be features such as the machine learning algorithms which will be released for SDS only. In the case of the machine learning algorithms, they execution occurs within the streaming engine but still relies on integration with the HANA database to persist the models.
As a general guideline we would recommend using SDS whenever the architecture includes HANA. ESP continues to be relevant in cases where HANA is not part of the environment.
Thanks
Rob