Busting two myths on In Memory Data Management
Fact: It is the need of the hour
Data is growing exponentially. With more automation, more structured data is being captured in our organizations. Also, with the widespread adoption of social media, views ,expressions and opinions are getting captured in Face book status messages, twitter tweets, Instagram’s pictures etc. which are categorized as unstructured data. The explosion of data can be summarized with the fact that the humans have doubled the amount of data that they produced in the last century in the last decade.
Nimble Organization analyzes information at faster speed to stay ahead of their competitors. For Example, Smart organizations are capturing their success of their product releases by analyzing the tweets and messages of its products real time and even making necessary course corrections. To make smart and faster analysis of the dataset so large and complex (which have been named Big Data for their sheer volume), having a faster system performance has become a basic hygiene factor of any system.
Disk Scan speed has only become faster by 10 times over the last 30 odd years. Compared to the data growth, it is not adequate and will add to more scan time. There is a requirement to remove the redundancy involved in scanning and moving the data from disk to CPU for processing. In Memory Data Management removes two time consuming processes to make the processing faster – Scanning the data from Disk and moving it to the main memory (DRAM). As the CPU have direct access to DRAM, the processing power is enhanced exponentially resulting in desired efficiency.
2. In Memory Data Management Concepts are not applicable in Enterprises
Fact: In Memory Data Management concepts have evolved over the years to serve enterprises
Typical Relational Database Management System (RDBMS) stores the data in convenient way so that Data can be easily written on it but scanning these records is time consuming. Also, RDBMS cannot be compressed easily, a basic requirement, to load more data in DRAM. Using Columnar Database makes scan faster and enables more compression but with additional drawback of write back to Database overhead. Due to extensive encoding used for compression, it is expensive to insert, update and delete records in columnar database.
Enterprises end up using both the databases – RDBMS system to write to the database (OLTP Systems) and Columnar DB to scan the DB for analytics (OLAP systems). Data movement across the database not only adds overhead to the system but also makes the data dated.
In Memory Data Management has evolved to cater to the Enterprise Needs. For Example, HANA, the In Memory Data Management Platform, has the columnar feature but performs only insert operations and avoids the time consuming update and delete operations. It segregates the data into passive data (data which have infrequent use) and active data ( regular/ recent data) for storage and operation optimization. Similarly, relevant modifications are done in the In Memory Data base to make it optimal for enterprise usage.
In Memory Data Management is the answers to today’s enterprise growing data computing demands. If optimized well, as seen in HANA, it can deliver amazing results such as analyzing F1 race at real time to de-codify the genome project much faster than before. Also falling memory prices and availability of additional processing power make In Memory Data the most suitable technology to invest for future. Would love to listen to your comments on this article.