This weekend I read an interesting article in Smart Data Collective (Intuitive Reasoning, Effective Analytics & Success : Lessons from Dr. Jonas Salk, SDC, 11-Apr-2015). As this was the 60th anniversary of the landmark vaccine that effectively wiped out Polio, the article was written to discuss how Dr. Salk's analytic approach was surely data-based - but more than that, his discovery grew out of an inquisitive and intuitive approach to studying data.
These days, we are now praising the once lowly data scientist whose obscure chi-square models and multivariate analyses are now expected to reveal the Holy Grail of ...well, whatever it is that we happen to be studying. That is, we think that if we can build a robust enough algorithm, the mystery of the ages can be revealed to us. This is simply poor analytic thinking. Typically, you start with a hypothesis founded primarily on your intuition and then you try to confirm or refute your hypothesis with hard data (reasoning). A lot of times, conventional wisdom (political thought, popular opinion, the view of our thought leaders) drives our analysis and we end up structuring data to support the popular view rather than challenging the herd mentality and looking for breakthroughs.
The history of SAP is filled with a lot of this unconventional thought - look at HANA. What was once something of a "fringe" idea (columnar data stores and massively parallel computing) has become mainstream - primarily due to the vision of a number of bright software engineers. Now this is becoming something of a mainstream idea. It was the intuition of people like Dr. Platter and others who are creating the breakthrough thinking that is propelling companies like SAP forward.
In your day to day analytic research, are you challenging the status quo and following your intuition? Or are you merely re-arranging the same chess pieces on the same chess board? Are you using visualization to challenge your intuition - more mostly looking for that perfect r-squared regression metric and the ideal straight line. Have you considered that the straight line that you are looking for is maybe a curve? Are you following your own intuition?
Have a look at this article and let me know what you think!!
