Events are the New Data

The Oxford Dictionary defines ‘Data’ as: “Facts … collected together”. In technical terms, ‘Data’ is more accurately: ‘Events folded together’ – ‘Folding’ being the merging of a particular Entity’s (State-changing) Events, in chronological order, to compute the latest Entity ‘State’.

Data – unlike Events – is an abstract notion. Whilst ‘Events’ represent actual changes to an Entity’s State (e.g. PurchaseOrderDeleted), ‘Data’ represents no more than the calculated State of a particular Entity, once all preceding Events have been folded together. At some point in history it was decided – owing to restrictive hardware costs at the time – that we could not possibly store all State-changing ‘Events’ in our Data-bases, but instead could store only ‘Data’: the ‘Folded State’ of each Entity (until such a time as even that was archived, owing to archaic hardware constraints).

The consequences of this historic Data-constraint are enormous. For one thing, it means that the Dimension of Time associated with all Events is thrown in the bin. For another, the intrinsic audit and logging potential of Events is also discarded. Finally, and perhaps more importantly given the exploding number of offline mobile use cases: the native ability of time-based Events to support OData Delta logic was completely lost (for which reason we scramble to reconstruct it post facto using complex mechanisms such as SAP’s Syclo Exchange Framework).

Whenever I hear such phrases as “Data is the New Oil”, it becomes clear that we remain trapped in our storage-constrained past, because what ought to be apparent, is that ‘Events are the New Data’. The very heavy investment of SAP in a new generation of ‘Column Store’ Data-base, is a mindless continuation of the same path first trodden in the 1960s; soon after our leap from tape to disk-based storage. Perhaps it is time for further reflection on this path?

Now that we have lost the shackles of our former storage constraints, we no longer have any logical, financial, or technical reason to be storing ‘Data’; we should instead be storing ‘Events’ which are, after all, the mother of all Data. Events should now all be persisted in today’s vast and cheap storage, and should only be folded into each Entity’s ‘Last State’ – In-Memory – after a certain delay (as proposed in my earlier Blog: How to build a ‘Rolling-Delta Database’ for Offline Mobile scenarios).

Using such a ‘Rolling-Delta’ mechanism would ensure that offline mobile scenarios could be fully managed In-Memory, whilst Events would nonetheless be folded as often as necessary in order to respect today’s physical Memory limits (cf. yesterday’s storage limits). Such an approach would add another dimension to our Analytical tools – and AI algorithms – by restoring the Dimension of Time to our persisted models. As such, what we should be discussing today is a next-generation, persistent ‘Event Store’ that completely replaces the Data-base, and an industry-standard In-Memory (Rolling-Delta) ‘Entity Store’, that correctly accounts for the fact that ‘Memory is the new Storage’.

‘Big Data’ can never represent any more than a subset of – Big – ‘Event Stores’, so let’s stop talking about Data (and Data Science, Data Architects, etc). Events are what matter, and today we have all we need to store and analyse them in real-time, whilst efficiently catering to the ever-growing demands of offline mobile applications; something we cannot properly support with 1960’s Data-based models (the very reason SAP dropped Client-Side Database ‘delta-tracking’ in SMP 3.0).