Real-Time Data Mining: In Memory Computing
With the introduction of high-speed, highly efficient computing platforms, it is now possible to query and analyze data faster than ever before. In-memory applications allow a customer to load its entire data set into active memory so that data can be analyzed in real time. This development greatly simplifies the data mining process and eliminates the need to schedule batch processes from a decentralized data warehouse environment.
This development, coupled with both retrospective and predictive analytics approaches allow a company to mine data and make decisions faster than ever before.
The biggest challenge for many companies is simply managing the volume of data being accumulated. According to IDC’s “The Diverse and Exploding Digital Universe 2010”, “the amount of information created and replicated will surpass 1.8 zettabytes (1.8 trillion gigabytes) – growing by a factor of 9 in just five years.”
IDC further states “In an information society, information is money. The trick is to generate value by extracting the right information from the digital universe – which, at the microcosmic level familiar to the average CIO, can seem as turbulent and unpredictable as the physical universe.”
From Data Management to Data Governance
Many companies have designed their IT structure to include Enterprise Data Management (EDM) principles to help ensure data quality. Unfortunately, simply managing the data does not necessarily result in the creation of actionable, strategic information. In essence, without proper discipline and structure, a company can arrive at the wrong answer more quickly than ever before.
Historically, EDM has focused on the “how” data is managed within an organization. This has largely been solely the responsibility of the IT department. EDM typically consists of structure around the following:
- Data Architecture – Where, when and how data is entered, stored, transmitted, reported
- Data Standards – Data definitions and schemas
- Data Governance – Data policies, ownership, roles and responsibilities
- Data Quality – Data metrics based on standards, process performance metrics
- Data Deployment – Business Intelligence integration, data process improvement
While these traditional EDM components are important, they fail to address the significant disconnect that often exists between the IT department and the rest of the business. This often results in challenges such as:
- An overabundance of reports that fail to provide meaningful insight into critical usiness decisions
- A proliferation of data sources (i.e. data marts, spreadsheets and desktop databases) that inhibit information sharing and create many versions of “the truth”.
- Poor adoption or lack of usage of IT department initiatives by the business
- Business needs not being met due to conflicting resource needs within the IT department.
The concept of Data Governance addresses these problems by wrapping the business needs around the traditional EDM components. The key to successful Data Governance is creating a team that provides thought leadership, with a deep understanding of the business as well as the IT solutions that are being leveraged.
This team should focus on performance areas outside of the traditional EDM components in order to create an overall data strategy that will lead to the creation, adoption and use of strategic information. Typical considerations for this team should include:
- Organizational Readiness – What is the current state of involvement of the business in the creation and consumption of data?
- Organizational Change Management – What is the plan for change management and communication in order to get the business thinking about strategic information?
- Predictive Analytics and Business Intelligence – What is the current state of analytics within the business? Is it based solely on retrospective analysis of data?
- Business Strategy – Where does the business want/need to go? What are their Key Performance Indicators? What data needs to be mined in order to accurately measure their progress toward meeting these goals?
- Competitive Differentiation – How does the business gain a competitive advantage through the use of strategic information?
- Fact-based Decision Making – How will strategic information be leveraged in the decision making process? Will management still rely on “gut feelings” to guide the organization or will they consult hard data to guide their decisions?
Coupling traditional analytics models with predictive analytics models and In-Memory computing can provide a real competitive difference in today’s business landscape. However, the successful implementation of these initiatives is heavily dependent on the organization’s readiness to adopt and actively use this strategic data.
What is the state of analytics within your organization today? Is data viewed as an asset that needs to be properly managed and protected? Or is your organization in need of an analytics renewal in order to ensure the judicious use of data to make your company a better-run business?