General News & Updates

thomashammer · ‎09-25-2023

Time is definitely flying: “Yesterday” we talked about innovations introduced in Q2, and now we can already proudly introduce new features delivered to SAP HANA Cloud, as part of our Q3 2023 release.

But before jumping into the details, let me share one thing:

In case you want to hear about the innovations from our product experts first-hand, make sure to join our What’s New in SAP HANA Cloud webinar scheduled for September 26, 2023. Register here!

Blog Contents

General News & Updates

Innovations in SAP HANA Cloud, SAP HANA database

Innovations in SAP HANA Cloud, data lake

Innovations in SAP HANA Cloud's Tooling

Further Innovations and Enhancements

General News & Updates

So, let's dive right into the innovations introduced with Q3 2023, and take a look at some general topic and enhancements.

Migration Self-Service

Our migration self-service represents a great mechanism for users to streamline the transition to SAP HANA Cloud from previous product generations of SAP HANA. In Q3, we have extended the functionalities offered by the service with great new capabilities.

Migration Self-Service - Current State

Availability on Google Cloud Platform

Before talking about the functional improvements, we are excited to share that the self-service as such is now also available on Google Cloud Platform. Which means that as of today, the offering is available on all four cloud service providers, including Alibaba Cloud, Microsoft Azure and Amazon Web Services.

Support for JSON Document Store of SAP HANA

The JSON Document Store in SAP HANA offers to store and process documents of the schema-flexible JSON format. As of now, the migration self-service is also capable to migrate data being stored in instances of the JSON Document Store of SAP HANA 2.0 instances (on-premise) to SAP HANA Cloud.

Support for HDI Container Data Migration

Similar to the JSON Document Store, the updated migration self-service for SAP HANA Cloud is now also able to migrate data from HDI Containers stored in instances of SAP HANA 2.0 (on-premise). With the functionality being available, tables inside HDI containers can be migrated together with tables in regular schemas. Nevertheless, please note that the service is not yet covering the migration of XSA service instances.

Innovations in SAP HANA Cloud, SAP HANA database

SAP HANA Native Storage Extension (NSE)

Data Tiering is an extremely powerful approach to optimize your data in the most cost-effective manner. With the SAP HANA Native Storage Extension, we are providing the capability to dynamically off-load data from SAP HANA’s in-memory storage to provide capacity for more frequently accessed and relevant (hot) data.

Intelligent Advisor for Storage Tiering

While offering great opportunities, incorrect data tiering decisions can also have significant impact on performance and user experience. Therefore, I am happy to announce the availability of the new Advisors for NSE, which will ease the setup of data tiering rules and configurations.

The NSE Advisor runs a rule-based algorithm on statistical information providing recommendations. It can be used to obtain/apply NSE recommendations to SAP HANA Cloud instances, to change the load unit of a table, partition or column to:

Page-loadable to reduce memory footprint without much performance impact

Column-loadable to improve performance by keeping hot objects in memory

Recommendations provided by the NSE Advisor

Doing so, you can achieve the best cost/performance ratio by determining which objects can be made page loadable, reducing in-memory footprint without significantly affecting performance.

The new NSE Advisor is an integrated tool within our SAP HANA Cloud Central and can be enabled with the click of a button. Once recommendations are derived, administrators can review the provided results and decide on how to apply them.

Configuration of the NSE Advisor

SAP HANA Deployment Infrastructure (HDI)

The SAP HANA Deployment Infrastructure (HDI) supports an efficient and reliable deployment of database artifacts to SAP HANA Cloud. Being widely used amongst our SAP HANA Cloud user base, we are proud to announce new features becoming part of the service.

Setup Remote Table Replications (RTR) via HDI artifacts

With the Q3 release of SAP HANA Cloud, users are now able to setup so-called remote table replications (RTR) via HDI artifacts. Via the integration of replication configurations into HDI, we are offering a very convenient mechanism for setup and enable the grouping with other HDI operations.

The connectivity setup and used via RTR is leveraging the Smart Data Access (SDA) technology.

Example: Replicate CUSTOMER table to target database

Via the illustrated example, you can see how easily the setup can be triggered and how the HDI operation is translated into SQL.

HDI syntax based on example

Multi-Model Data Processing

The Multi-model data processing capabilities for SAP HANA Cloud are a core value of our SAP HANA database. With the new release, we have once again strengthened the integration between the various data processing engines, specifically now for graph and spatial analytics.

Geo-Spatial Analysis on Graph

With the QRC3 2023 update, SAP HANA Cloud's multi-model area will introduce support of spatial types and functions in GraphScript language. With this the GraphScipt will support all methods of ST_GEOMETRY that are listed in the SAP HANA Cloud, SAP HANA Database Spatial Reference, which do not return ST_GEOMETRY or VARBINARY data types as outputs. This capability will empower GraphScript algorithms to evaluate the spatial characteristics of graph vertices and edges. Such functionality could prove valuable in a range of scenarios, including identifying optimal routes, avoiding restricted areas when determining shortest paths, dynamically calculating edge weights within a graph network, and various other applications.

Machine Learning and Predictive Analysis

SAP HANA Cloud’s embedded machine learning and predictive analysis capabilities help to build intelligent data applications without the need for secondary systems and extensive pipelines.

In the area of AutoML, the SAP HANA Cloud Q3 2023 release adds regression operator support for enriched AutoML time series scenarios (see all innovations). Furthermore the Predictive Analysis Libraries’s (PAL) adds an improved one-hot encoding method of categorial input features significantly saving runtime resources and adds a new quantile transformation method to achieve uniform data distributions during pre-processing of data. The Hybrid Gradient Boosting Tree (HGBT) regression algorithm supports a new objective function Huber loss, yielding in more robust regression models regarding outliers and noise in data.

In addition to that, the long-term time series forecasting (LTSF) function supports a time series decomposition function for the neural network time series models. The time seasonality and trend test function has been enhanced to be less sensitive to outliers in the time series data.

The Python machine learning client 2.18 adds support for new PAL algorithms and various new visualizations like SHAP explainer for AutoML scenarios or the regression model report adding sore-output charts. The enhancements list in detail can be found here.

Graphical View Modeling

As usual, we are also very happy to share some details on the enhancements and new capabilities introduced for graphical view modeling via so-called calculation views.

MDS Cubes

First, we want to introduce the all-new MDS Cube. MDS Cubes represent submodels of calculation views, which are explicitly loaded via an API so that materialized data can be queried. Data in MDS Cubes is stored in a highly-optimized way. This optimized data structure can be leveraged in analytical reporting that is based on MDS metadata, such as SAP Analytics Cloud. The described MDS Cubes are a great new option to speed up analytical queries while reducing resource consumption at the same time.

MDS Cube as submodel of calculation view (example)

It's important to bear in mind that data in MDS Cubes reflects the state at the time the MDS Cube was loaded. Only data which was visible to the user who loaded the MDS Cube is included. Loaded data is not automatically updated. This can lead to differences when comparing reporting based on MDS Cubes and based on the online calculation view. To reflect data changes in reporting, it is necessary to manually re-load the MDS Cube.

Please note, that if a calculation view feature is not supported by MDS Cubes, the tab 'MDS Cubes' is inactive, and a list of feature(s) is displayed that prevents the usage of MDS Cubes. Removing these feature(s) from the calculation view activates the 'MDS Cubes' tab.

Moreover, if an MDS Cube is already defined in a calculation view, features of the calculation view that are not supported by MDS Cubes are deactivated. When hovering over the feature you will be informed that it is inactive due to the existence of an MDS Cube definition. If the MDS Cube definition is deleted, the feature will become available again.

Propagate Column Name Changes

In order to change the name of a column throughout the calculation view stack within the current HDI container the new option “Propagate recursively in Consuming Views” of the “Rename & Adjust References” dialog is a useful tool.

Propagate Column Renaming

With the the option being checked, the column will be renamed recursively throughout the calculation view stack.

Potential warnings, such as that a mapped column name will be overwritten, are shown:

Warning during column renaming

Be assured that renaming operations, which would lead to conflicts (e.g. non-unique column names), will be aborted and an error message will be presented:

Error during column renaming

If the option is unchecked, only the input columns of directly dependent calculation views are renamed.

Thanks to the new renaming option, we hope to simplify refactoring efforts for our users significantly.

Where-Used Functionality

QRC3 2023 has also introduced additional features to gain more insights into column usage, to ease refactorings:

You can use the new 'Where Used Inside Calculation View' functionality to understand where a certain column is used in the current calculation view:

Where Used Inside Calculation View

Similarly, you can use the 'Where Used Outside Current View' to understand the usage of a column, input parameter or variable in other objects inside the same HDI container:

Where Used Outside Current View

The dependencies within the same HDI container will then be listed:

Output Example: Where Used Analysis

Restrictions Based on Measures

Restrictions of restricted columns can now also be placed on measures. This can be used for example, to filter out outlier values before aggregation.

Define Restrictions on Measure

Graphical support of user-defined functions with multiple return values

The expression editor now displays the parameters which are returned by a function if multiple values are returned. In addition, you can now select which parameter value should be used in the expression:

Example: User Defined Function with multiple return values

Flexible Specification of T009 and T009B data during fiscal calendar data generation

When generating fiscal calendar data, it is now possible to specify tables or synonyms which contain (or refer to) the data that would typically be expected to be stored in the T009 and T009B tables. The structure of the target tables have to match the structure of the original T009/B tables:

Example: Data Generation Fiscal Calendar

This new feature provides more more flexibility and makes re-use of table data easier.

Innovations for SAP HANA Cloud, data lake

There are many extremely important behind the scenes changes happening to improve the performance and availability of the SAP HANA Cloud, data lake service, which are transparent to end-users.

However, we do have a couple of very nice improvements that will make your life easier when using the SAP HANA Cloud data lake.

Wildcard Support for LOAD TABLE

The 'Wildcard Support' represents a feature many users have been looking forward to taking advantage of. Up to now, users have had to name each file individually when loading data from files into the data lake relational engine. This means lots of typing or scripting, both error prone processes, especially when loading large numbers of files, which is a common requirement.

With this feature, you can simply use the wildcard character ‘*’ to represent any sequence of characters in a filename (*) when loading data from data lake Files or external cloud provider storage. You can also use a double wildcard (**) to represent a directory to give you more control over a set of files you wish to load. Of course, you can combine these characters to create custom matching strings to suit your needs.

Please note that even though wildcards are now supported, a single LOAD statement is still limited to loading 10,000 files.

Also note that when loading from a cloud provider location, you cannot use a wildcard in the initial prefix for the set of files you wish to load (ie. the ‘bucket’ name).

File Based Access Controls

The second feature I want to highlight is providing benefits for the data lake files component of SAP HANA Cloud.

Currently the only way to have some control over file access in data lake Files is to use the home directory feature, which allows users to place files they do not wish other users to have access to in their user home directory. This is a limited form of access control, and makes it difficult to manage security for data lake Files.

However, with the new fine-grained access control for files, you can specify (or deny) read and write access to individual files and directories in data lake Files.

The access control list (ACL) is controlled via a REST API call using a JSON document to define the access permissions on a file or directory for a user or a role in a data lake Files instance.

For now, to set access controls, you must use the REST API directly, but tooling support for the capability will be provided soon.

Innovations in SAP HANA Cloud’s Tooling

SAP HANA Cloud offers powerful tools to efficiently work with our cloud database service. To make these tools even stronger, we have introduced great new capabilities.

Innovations in SAP HANA Cloud Central

SAP HANA Cloud Central represents SAP HANA Cloud’s Swiss army knife, providing tools to manage your entire SAP HANA Cloud landscape, but also to monitor and administrate single instances. With the addition of a SQL console, we have even introduced capabilities for Developers and Data Stewards.

New SQL console functionalities

The recently released SQL console in SAP HANA Cloud Central has also received some great new capabilities with the latest update of SAP HANA Cloud. With the new additions and enhancements, we are improving our users productivity and making the tool even more feature rich.

These new features include:

Ability to download or copy structured query language (SQL) results and specify the format setting (CSV or fixed width)

Visibility of the current schema and an option to set the schema

Additional SQL console tab settings enabling the specification of:

Ability to support the theme set defined for SAP HANA Cloud Central

New SQL console functionalities

New functionality in the SAP HANA Database Explorer: Import of Delta Lake and Hive partitions and more

Additionally, we have also introduced new capabilities to our SAP HANA Database Explorer, which remains a very important tool when working with SAP HANA Cloud.

With QRC3 2023, SAP HANA Database Explorer is supporting the import of files from a directory:

Import multiple CSV files from one directory

Import parquet-formatted files in an Apache Hive partition or Delta Lake directory

Import from multiple CSV files and parquet files from Apache Hive/Delta Lake

Moreover, we are offing the option to select one or more files or folders for deletion in a data lake files connection, to run operations more efficiently.

Multi-file select for deletion - Data Lake Files

Further Innovations and Enhancements

SAP HANA Workload Management

Default Parallelism of Single SQL Statement Execution

With QRC3 2023, a new default value has been introduced to avoid very high parallelization of a single SQL statement. It is applied automatically when provisioning a new SAP HANA Database instance in SAP HANA Cloud or upgrading to QRC03 2023 or higher. In case you have set a custom value for configuration parameter global.ini [execution] default_statement_concurrency_limit (_rel), no changes will be noticed.

Benefits:

The new default-value avoids a very high parallelization of a single SQL statement in the SAP HANA Database instance. It ensures that at maximum 25% of the available computation resources of a SAP HANA Cloud instance can be used by a single SQL statement

Details:

The default-value can be changed by customers, but SAP recommends to set up Workload Classes for each application by using the properties “TOTAL STATEMENT THREAD LIMIT” and / or “STATEMENT THREAD LIMIT”. For more details, please also see SAP Note 3311477

User-Management and Authorizations

IP- and application-based connect restrictions for user groups

As our customers demanded more fine-granular control over security configuration settings e.g. to be able to implement Segregation of Duties (SoD) concepts, we have introduced IP- and application-based connection restrictions for user groups to the SAP HANA database within SAP HANA Cloud.

Both capabilities enable customers the flexibility to limit access for all users or parts of their user base, for instance, to permit administrators to log in from the customer's IP range only (released in Q2 2023) and application range based on the application session variable (released in Q3 2023).

The additional security features provide more fine-granular control over security configuration settings, such as to be able to implement SoD concepts, and thereby help customers to further secure their systems against potential attacks from the internet.

Data Transport Security

Support for TLS 1.3 for SAP HANA Database (SQL Connections)

TLS 1.3 is, in some environments, required for data transport security. With the latest update of SAP HANA Cloud, we can satisfy this requirement, as with the Q3 Version of SAP HANA Cloud, the SAP HANA database supports TLS version 1.3 for SQL connections.

To leverage TLS1.3, it is not necessary to use any particular version of the SAP HANA Clients, however, the security libraries that are (automatically) used by the clients need to fulfill certain version requirements:

CommonCryptoLib (CCL) >= 8.5.48 or

OpenSSL >= 1.1.1

Looking for more details on the latest features and updates to SAP HANA Cloud?

Why don’t you take a look at our What’s New Viewer in our Technical Documentation, which describes the entire scope of the release.

In addition, to get first-hand demonstrations on the latest innovations in SAP HANA Cloud, and to be able to bring up any of your open questions, you can register for our “What’s New in SAP HANA Cloud“ webinar, which is planned for September 26, 2023.

My colleagues from the SAP HANA Cloud product management team and I will share more details about the release highlights.

Finally, it’s important to say: Don’t forget to follow the SAP HANA Cloud tag to not miss any updates on SAP HANA Cloud! In addition, the whatsnewinsaphanacloud tag collects all what’s new blog posts, so you won’t have a hard time finding all the necessary information. In the unfortunate case that you have missed the What’s New webinar in Q2 2023, you can find it and all future webinars in this playlist on YouTube.

Do you want to discuss some of the outlined innovations or have any other questions related to SAP HANA Cloud? Feel free to post them in our SAP HANA Cloud Community Q&A or in the comments below.