Will Big Data create a Shallower Gene Pool? (subtitle: a forearming in favour of data diversity)

Sitting in the Beyond Retail 2013 conference just now and listening to a presentation on customer analytics in the Retail industry and it reminded me of an unpublished blog I wrote a while back. Probably not appropriate in this context, and I will publish elsewhere in time, but for now thought I’d get it ‘off my chest’ here. Apologies in advance for the self indulgence …

So I work in the Information Technology industry, and specifically in the Data Warehousing world. I have a professional interest in keeping abreast of the latest IT developments; and the latest IT trends indicate that much, much more (structured) data will become available to us all in the not too distant future … but perhaps the full depth and breadth will be available to only some in our society. Data is being stored persistently at an ever-increasing level of granularity and being manipulated by increasingly more sophisticated algorithms. Where will it end?

It got me wondering … what if data was like DNA?

What if data diversity was one of the things that protected us all from catastrophic demise from a ‘left-field’ or ‘black swan’ event (pardon my mixed metaphors)

Take the financial crisis of 2008/2009: it’s a good, if over-quoted, example. There, the pricing of risk in the financial economy did not sufficiently represent the true risks in the real economy. And the resulting meltdown occurred due to some ‘incestuous’ relations within a small group of financial institutions.

But what if the trend towards global economies of scale coupled with the rise of big data created an evolutionary cycle similar to the 2008/2009 financial crisis where fewer autonomous ‘decision making hosts’ exist, and the diversity of data storage and, more importantly, the diversity of interpretation was diminished?

Wouldn’t we all then become exposed to a common set of threats, again?

I’m thinking about insurance companies using vast datasets to ‘reward’ low risk behaviour, or retailers mining our buying habits to maximise global economies of scale which may effectively squeeze out local culture-specific buying habits; or perhaps the ethical challenge of health insurance providers monitoring our genetic make-up to prohibit families with a predisposition to known ‘expensive’ conditions from procreating.

Under these conditions the single human trait that has done most to ensure our survival as a species, namely adaptability, may well wither if it becomes increasingly more difficult for mankind to stray from the ‘beaten track’, the laws of collectivism being as strong as they are.

Moreover, could the new terrorism come perhaps from privacy advocates and those seeking to break the hegemony of the soon to be established ‘datocracy’ – my own application of the term for the system of organisation where control over data (and its interpretation) is the supreme power (I was gutted find out that there was already a website)

Will we be signing the death warrant of the species if we allow control of data to vest in a select-few. Maybe we should begin to actively protect the diversity of data and the interpretation of data trends and guarantee open access to all such results?

Maybe this is nothing more than just an argument against censorship? And isn’t it then just a young person’s argument? After all middle-of-the-road, mid-life mediocrity is better served by smaller peaks and shallower troughs; by less dramatic winds of change.

… but can we afford to take that risk: what if a new socio-economic regime rose up, called “Informationism”, wherein those who controlled the means of making decisions also controlled our societies, controlled our economies and ultimately our future generations.

Dunno, all I know is that I need more data to back up my hypotheses …