The dilemma at the heart of a series of recent blog posts on data tiering is familiar to many organizations: how to maximize data value, accessibility, and security while minimizing the costs of storing endlessly accumulating information.
Data tiering (read a comprehensive overview here) offers a straightforward, elegant solution to the increasingly vexing problem of data storage: scale and score data so that it can be stored with the solution that offers the best cost to performance ratio. Frequently accessed, high-value data is categorized as “hot” and stored in-memory on the SAP HANA database. Less frequently accessed “warm” data is stored on a lower-cost storage tier managed as a unified part of the SAP HANA database. And, finally, rarely accessed, low-value data is stored on low-cost storage tiers like disk or Hadoop, which remain accessible at any time.
To manage these different temperature categories of data, SAP customers have a number of different tiering options, dependent on the type of application in use. What follows is an overview of those options for SAP HANA native applications; SAP BW on HANA and SAP BW/4 HANA; and Suite on HANA and SAP S/4 HANA.
Hot Data Tiering
As denoted in the diagram above, all the applications share the same in-memory storage capabilities in SAP HANA for mission-critical data that requires real-time access. Non-volatile memory (NVM), also known as SAP HANA persistent memory (PMEM), adds additional storage capacity for data held in-memory. PMEM offers lower TCO and faster database start-up time after a shutdown than DRAM, but is not a replacement as DRAM is still required for SAP HANA systems.
Warm Data Tiering
When data becomes less mission critical and moves from hot to warm, native SAP HANA, SAP BW on HANA, and SAP BW/4 HANA offer a common warm storage tier option: the Extension Node, with nearly in-memory performance and full SAP HANA functionality (a previous post offers a detailed exploration of the Extension Node). The SAP BW on HANA modelling tool and SAP BW/4 HANA Data Tiering Optimization (DTO) manage the classification and relocation of data between SAP HANA in-memory and Extension Node.
Native SAP HANA applications have an additional warm data storage option, which offers lower costs and similar performance to the in-memory Extension Node, called Dynamic Tiering. Dynamic Tiering is a disk-store option that is part of the SAP HANA landscape and is accordingly easy to implement and manage but does not support certain SAP HANA capabilities, such as geospatial and time series functions. (Take a closer look at Dynamic Tiering for warm data management).
For native SAP HANA applications, the Data Lifecycle Management (DLM) tool, which is part of the SAP HANA Data Warehousing Foundation (DWF), facilitates warm data tiering by enabling rule-based movement of data between SAP HANA in-memory and Dynamic Tiering or Extension Node.
Suite on HANA and SAP S/4 HANA implement Data Aging for warm data by enabling Paged Attributes within SAP HANA. Paged Attributes are only applicable in the context of Data Aging. For Suite on HANA, Data Aging requires the implementation of aging objects in Suite Applications. Only the documented aging objects listed in the online help section for Suite on HANA or SAP S/4HANA can be moved between SAP HANA in-memory and disk. Periodic aging runs are executed within the Data Aging framework that take into account the data of aging objects paged on disk. During the aging run, all tables of an aging object are partitioned, in a part, with current “hot” data and historical “warm” data. All historical partitions of a table are marked as page-loadable, which means the columns of historical partitions are defined as Paged Attributes.
Paged Attributes offers a targeted warm tier solution, allowing users to load and unload table columns page by page from the SAP HANA persistence layer into memory as needed, avoiding the added burden of loading entire columns of tables at once. Additionally, and even more importantly, Paged Attributes receive a high unload priority from the SAP HANA database: they are paged on disk first when SAP HANA decides to free up memory. (Find out more about Paged Attributes.)
Cold Data Tiering Options
Just as all hot and warm data tier storage is housed on the SAP HANA database, all cold data tier storage exists on external stores, such as SAP IQ, SAP HANA Vora or Hadoop. As we already know from warm data tiering, applications have different SLA requirements and therefore offer their own cold storage tiering approach. And the good news? Cold data tiering tools are mostly the same as what’s used for warm data tiering. (Find an in-depth look at cold data tiering here.)
Native SAP HANA applications use the DLM tool to support Hadoop or cloud storage destinations. DLM facilitates data tiering by enabling a rule-based and seamless movement of data from hot or warm storage to the appropriate cold data store, and vice versa. (Check out this DLM overview or examine DLM in technical detail.)
SAP BW on HANA and SAP BW/4 HANA manage the classification and relocation of data for read only and sporadic reporting access with Data Tiering Optimization (DTO) or Near-Line Storage (NLS) for external storage in SAP IQ. If using Hadoop or similar systems for external storage, DTO—in conjunction with SAP Data Hub, SAP HANA Vora or the SAP HANA Spark Controller—is used to manage cold data. (Learn more about using DTO for cold data tiering)
Finally, Suite on HANA and SAP S/4 HANA users can archive cold data on external stores with the ILM Store, an ABAP solution that creates archive files using SAP NetWeaver as the application server. In order to read back these archive files, for auditing purposes for example, the files must be converted and imported with ILM. Regardless of this extra step, the ILM Store reduces the data footprint in SAP HANA by allowing for cold data storage on the less costly SAP IQ database system and external solutions like Hadoop. ILM specifically provides a Hadoop Connector to store archive data files to HDFS (Hadoop Distributed File System). The connection parameters are maintained in the ILM Store to establish the communication with the Hadoop instance. (Find more on ILM)
Comprehensive Data Tiering Options
The comprehensive approaches to data tiering detailed within this post and throughout our Data Tiering blog series provide a number of data storage and access options, each designed to provide SAP HANA customers with maximum data value at the minimum cost. We’d be happy to answer any questions you may have about any of the outlined SAP Data Tiering recommendations—simply contact us directly via email or use the comments section below and we’ll get back to you immediately.
Data tiering (read a comprehensive overview here) offers a straightforward, elegant solution to the increasingly vexing problem of data storage: scale and score data so that it can be stored with the solution that offers the best cost to performance ratio. Frequently accessed, high-value data is categorized as “hot” and stored in-memory on the SAP HANA database. Less frequently accessed “warm” data is stored on a lower-cost storage tier managed as a unified part of the SAP HANA database. And, finally, rarely accessed, low-value data is stored on low-cost storage tiers like disk or Hadoop, which remain accessible at any time.
To manage these different temperature categories of data, SAP customers have a number of different tiering options, dependent on the type of application in use. What follows is an overview of those options for SAP HANA native applications; SAP BW on HANA and SAP BW/4 HANA; and Suite on HANA and SAP S/4 HANA.
Hot Data Tiering
As denoted in the diagram above, all the applications share the same in-memory storage capabilities in SAP HANA for mission-critical data that requires real-time access. Non-volatile memory (NVM), also known as SAP HANA persistent memory (PMEM), adds additional storage capacity for data held in-memory. PMEM offers lower TCO and faster database start-up time after a shutdown than DRAM, but is not a replacement as DRAM is still required for SAP HANA systems.
Warm Data Tiering
When data becomes less mission critical and moves from hot to warm, native SAP HANA, SAP BW on HANA, and SAP BW/4 HANA offer a common warm storage tier option: the Extension Node, with nearly in-memory performance and full SAP HANA functionality (a previous post offers a detailed exploration of the Extension Node). The SAP BW on HANA modelling tool and SAP BW/4 HANA Data Tiering Optimization (DTO) manage the classification and relocation of data between SAP HANA in-memory and Extension Node.
Native SAP HANA applications have an additional warm data storage option, which offers lower costs and similar performance to the in-memory Extension Node, called Dynamic Tiering. Dynamic Tiering is a disk-store option that is part of the SAP HANA landscape and is accordingly easy to implement and manage but does not support certain SAP HANA capabilities, such as geospatial and time series functions. (Take a closer look at Dynamic Tiering for warm data management).
For native SAP HANA applications, the Data Lifecycle Management (DLM) tool, which is part of the SAP HANA Data Warehousing Foundation (DWF), facilitates warm data tiering by enabling rule-based movement of data between SAP HANA in-memory and Dynamic Tiering or Extension Node.
Suite on HANA and SAP S/4 HANA implement Data Aging for warm data by enabling Paged Attributes within SAP HANA. Paged Attributes are only applicable in the context of Data Aging. For Suite on HANA, Data Aging requires the implementation of aging objects in Suite Applications. Only the documented aging objects listed in the online help section for Suite on HANA or SAP S/4HANA can be moved between SAP HANA in-memory and disk. Periodic aging runs are executed within the Data Aging framework that take into account the data of aging objects paged on disk. During the aging run, all tables of an aging object are partitioned, in a part, with current “hot” data and historical “warm” data. All historical partitions of a table are marked as page-loadable, which means the columns of historical partitions are defined as Paged Attributes.
Paged Attributes offers a targeted warm tier solution, allowing users to load and unload table columns page by page from the SAP HANA persistence layer into memory as needed, avoiding the added burden of loading entire columns of tables at once. Additionally, and even more importantly, Paged Attributes receive a high unload priority from the SAP HANA database: they are paged on disk first when SAP HANA decides to free up memory. (Find out more about Paged Attributes.)
Cold Data Tiering Options
Just as all hot and warm data tier storage is housed on the SAP HANA database, all cold data tier storage exists on external stores, such as SAP IQ, SAP HANA Vora or Hadoop. As we already know from warm data tiering, applications have different SLA requirements and therefore offer their own cold storage tiering approach. And the good news? Cold data tiering tools are mostly the same as what’s used for warm data tiering. (Find an in-depth look at cold data tiering here.)
Native SAP HANA applications use the DLM tool to support Hadoop or cloud storage destinations. DLM facilitates data tiering by enabling a rule-based and seamless movement of data from hot or warm storage to the appropriate cold data store, and vice versa. (Check out this DLM overview or examine DLM in technical detail.)
SAP BW on HANA and SAP BW/4 HANA manage the classification and relocation of data for read only and sporadic reporting access with Data Tiering Optimization (DTO) or Near-Line Storage (NLS) for external storage in SAP IQ. If using Hadoop or similar systems for external storage, DTO—in conjunction with SAP Data Hub, SAP HANA Vora or the SAP HANA Spark Controller—is used to manage cold data. (Learn more about using DTO for cold data tiering)
Finally, Suite on HANA and SAP S/4 HANA users can archive cold data on external stores with the ILM Store, an ABAP solution that creates archive files using SAP NetWeaver as the application server. In order to read back these archive files, for auditing purposes for example, the files must be converted and imported with ILM. Regardless of this extra step, the ILM Store reduces the data footprint in SAP HANA by allowing for cold data storage on the less costly SAP IQ database system and external solutions like Hadoop. ILM specifically provides a Hadoop Connector to store archive data files to HDFS (Hadoop Distributed File System). The connection parameters are maintained in the ILM Store to establish the communication with the Hadoop instance. (Find more on ILM)
Comprehensive Data Tiering Options
The comprehensive approaches to data tiering detailed within this post and throughout our Data Tiering blog series provide a number of data storage and access options, each designed to provide SAP HANA customers with maximum data value at the minimum cost. We’d be happy to answer any questions you may have about any of the outlined SAP Data Tiering recommendations—simply contact us directly via email or use the comments section below and we’ll get back to you immediately.
- SAP Managed Tags:
11 Comments
