In 2013, SAP IQ 16 introduced a new feature called the RLV (Row Level Version) in-memory store. The RLV store gave IQ users the ability to perform concurrent, high speed writes to the same IQ table. The motivation for developing a write-optimized delta store came from the Big Data craze, where not only volume and variety, but velocity of data defines today’s digital climate. Data is streaming at unprecedented speeds, and the ability to react quickly is imperative in a competitive business environment. For an analytics database like IQ, it is critical not only to execute fast query processing, but also to ingest and make operational data available in near real time. IQ’s traditional bulk loader is highly parallel and efficient, but it is designed for large batch updates to minimize the time the IQ table is write locked.
The RLV store augments the IQ main store with a write optimized in-memory delta store. It has “write friendly” characteristics, such as row level locking, append-only updates, reduced compression, no sorting, and lightweight indexing. The user has the flexibility to choose which tables are RLV enabled for low latency concurrent updates. Traditional, “non-RLV” IQ tables reside completely in the main store as always, with table level locking and bulk data loading.
Periodically, the RLV in-memory store merges its changes into the IQ main store on disk, to consolidate data and free up RLV memory for further activity. The merge becomes the single writer to the IQ main store and amortizes the cost of updates while the RLV in-memory store permits concurrent updates. Although in flight RLV data is resident only in memory, the RLV store is fully recoverable with its own dedicated transaction log. On the query side, the IQ query engine accesses both stores at the same time through a “consistency view”, and is able to return query results on all the latest data:
So, why are we talking about RLV again?
The reason is that IQ technology is being incorporated into the SAP HANA database for management of large volumes of less frequently accessed, or “aged” data. This enhancement to SAP HANA is called “HANA dynamic tiering”, and SAP BW is making use of this capability for large data warehouses. Key to SAP BW’s implementation of dynamic tiering is IQ’s RLV store, which allows BW to ingest concurrent streams of data directly into the dynamic tiering “warm store”. The benefit to the user is a reduced HANA footprint, where hot data is maintained in HANA memory, and cooler data lives in IQ’s disk-backed column store.
During the course of implementing HANA dynamic tiering, the SAP BW team put the RLV store through rigorous functional and stress testing. This resulted in quality improvements and several new feature requests. The RLV store is now better than ever, with the following enhancements:
- A fine-tuned auto-merge algorithm that is more sensitive to data changes in the RLV store
- Better garbage collection of deleted rows in the RLV store
- Enhanced memory management to reduce the size of the RLV store
- A simple merge procedure that can be invoked by the user to explicitly merge data for all RLV tables into the IQ main store in a single operation
So say “Hello again” to the RLV store, to bring the power of high velocity data streaming into your IQ database.
You can read more about RLV here: