All humans thirst for knowledge. This feature of our species is highly adaptive, hard wired, and is the byproduct of evolution. Although we’re born with certain skills and abilities, we learn through experience and acquire knowledge. Kahneman and Tversky postulated in their seminal work in Psychology and Economics that humans have two systems for acquiring knowledge. They characterize them as follows:- System 1: Automatic, Fast, Frequent, and Subconscious
- System 2: Slow, Effortful, Infrequent, and Conscious
System 1—Our Instinctive Defense
Every day we use System1 to make numerous decisions or predictions on how the world works. When walking across the street we scan the environment to identify risks and estimate if it safe to proceed. These are all predictions that happen automatically using System 1. We aren’t born with this knowledge—we learn it via trial and error and operationalize these models into production.
When asked about how we make these decisions, we often don’t know how to explain how we make decisions, but we trust them implicitly because we have learned they work. Only when these models or heuristics fail too frequently are we forced into using System 2 to develop an alternative approach.
System 2—Our Deliberate Defense
System 2 requires effort to make sense of the “data” we encounter in our life. The models that we derive from System 2 require statistical inference and data. When we are smart enough, collect enough data, and use a scientific approach to inference, some of us are able to derive “first principle knowledge.”
For example, before the seventeenth century, we had beliefs about how the solar system worked—some of which postulated that gods pushed the planets across the sky. While we had no proof, this belief could explain in a naïve way how the solar system appeared to work. Eventually the work of astronomers led to Kepler’s law which allowed for repeatable predictions. Kepler’s law led to Newton to come up with the law of gravity, which is perhaps our most important example of First Principal Knowledge. So the process of moving from System 1 to System 2 starts with Beliefs, moves to Empirical insight, and (if we are lucky and smart enough) we uncover First Principal Knowledge.
Overcoming the Flaws in the Systems
Kahneman and Tversky’s research showed that both systems are flawed. System 1 works but is subject to numerous biases which cause us to make mistakes. System 2 is also flawed because, in general, humans are poor at statistical inference. Apparently the best way to overcome these shortcomings inherent in how we learn is to develop reuseable programs that will learn for us. Enter the age of machine learning.
What Has Changed?
The information age has forever changed our world because we now double the amount of data from which to derive knowledge every two years. The combination of sensors, Big Data management techniques, and machine learning has empowered us in a unique way.
As the explosion of data continues to accelerate, it’s too much to deal with for traditional approaches to knowledge discovery. The amount of data we have available to us is really too much for us to use traditional approach of generating hypothesizes, running experiments, collecting and analyzing data. The scientific method will need machine learning to help us conduct the experiments to uncover new First Principal Knowledge.
In short, machine learning is going to simplify and improve how we use System 2. Buckle up. It is going to be an epic ride.
- SAP Managed Tags:
