Forget about “Big Data”, forget about “IoT”. Think Optimization
A funny thing happens when you come up with a label for a particular concept. Suddenly something that had perhaps a fuzzy meaning or couldn’t easily be thought or talked about before becomes a “thing” which allows us think about it, discuss it, even build businesses around it. But the very act of labeling can also obscure and drive thinking along a particular path that may lead us into dead ends and prevent us from truly grasping the significance of the concept or its wider context.
I would argue, this has happened to the terms “Big Data” and “Internet of Things (IoT)” and the consequence is that we’re looking for Big Data and Internet of Things use cases, and as a result flailing about, expending energy, flapping our wings, but in many cases getting nowhere. We may by now be convinced that we have to do something with Big Data and IoT and that it holds much promise. We may even be scared to miss the boat and that our competitors are further along, and that if we don’t start now we will fall further and further behind.
I was talking to a customer the other day, and the conversation started off almost typical: we know we need to do something around Big Data, we have access to large datasets, so what architecture do we need? My response was, that it depends, what is it that you are you planning to do? “Well”, was the reply, “we plan on putting together a data lake so that the business can ask any questions they would like”. That is, the approach to Big Data was entirely technical, and no real thought had yet been put into why or for what reason we might make the investment for a Big Data landscape in the first place. The almost inevitable result is failure, and articles like this and this.
And to be honest, this is an easy trap to fall into. I myself have been looking for Big Data and IoT use cases, and trying to think where best to apply its techniques, and found myself struggling. I started to think about “Smart Data” instead of Big Data, to at least reflect that Big Data requires advanced/predictive analytics to be truly useful, and that helped widening the scope a bit, but in the end didn’t really solve the problem: because I was still thinking in terms of the technology, rather than where the technology should be applied.
“Big Data” tells us we’re going to analyze large data sets. We might use “unstructured data” (even though the definition of unstructured data runs from CSV files all the way to images, video and audio). We find ourselves falling back on social media feeds as an obvious default, regardless whether it may or may not be appropriate to our use case or business. We may even argue endlessly whether there are 3, 4 or 5 “V”s, as if that is somehow important or relevant to the problems we’re trying to solve.
“IoT” tells us we’re going to use internet connected devices. We’ll use the measurement of sensors to understand and predict…. something. But what? The German term “Industrie 4.0” is a term I like a little better, as at least it has a bit more of a focus towards corporations, in particular manufacturing. But even there it seems the concept drives the discussion, rather than how it enables us to run better.
These concepts, therefore, are mental traps. They force us to start with the technology, rather than where we might apply the technology in the most optimal way. It’s the classical “solution in search of a problem”. Whether we start with the concept, or we start with the data set (“I have this fantastic large data set, what wonderful things is it able to tell me?”), we are simply starting from the wrong place. And because Big Data and IoT are still a bit fuzzy and undefined, and we lack focus, anything becomes possible, and we’re likely to overshoot and be overambitious if we do identify a “Big Data” or “IoT” use case, and fail to appreciate what we may already have.
I propose, therefore, that we think of it in a different way: What if we simply thought of it as an opportunity to optimize business processes? That is, we pick something that we already do inside the business, and use new techniques to improve them, optimize them. That leads us to Manufacturing Optimization, Supply Chain Optimization, Human Resource- or Human Capital Optimization, Customer Relationship Management Optimization, or more generically, Business Process Optimization.
This is not particularly profound. It is really just a mental switch you make to think about the problem in a different way. Rather than looking for use cases for IoT and Big Data, we’re looking for use cases to optimize an existing business process. That provides us with immediate focus, first by shrinking the domain space (I like parameters and constraints; they limit the field of investigation immediately to something much smaller and more concrete), and then by focusing our search for datasets and techniques that can assist us in our goals. If we’re trying to improve a manufacturing or supply chain business process, for instance, we probably don’t need a Twitter or Facebook feed, but might look for particular sensors or location-aware devices instead.
It also forces us to re-appreciate the data we already have, and whether we’re leveraging it appropriately. In Customer Relationship Management, for instance, corporate email is still often an underutilized resource, while it can just through metadata analysis already provide a list of everyone in the company who touched a particular customer. And of course, there is a wealth of information in our ERP systems and data warehouses (BW or otherwise) that could tell us a lot more than weblogs or mobile device location data ever could. Such data, for instance historical weather reports in Retail, can certainly be used to enrich existing transactional data, but such data by itself is largely going to be meaningless or at least not as effective as when combined with data we already have.
Once we know what we want to optimize, and have identified the data sources required for it, the definition of the solution is going to be much easier, much more concrete, and with a much clearer justification for investment and ROI calculation. And since S/4HANA allows us to customize and add to HTML5 front-end applications, we can bring the results of our solution directly into those applications where it is most clearly needed and provides the most business benefit.
In conclusion, then, forget about Big Data and IoT, not because these are not important techniques. They are, and their promise and potential is enormous. But let’s think of optimization, focus ourselves on the business problem we’re trying to solve, and only then figure out how to apply Big Data and IoT techniques to achieve that goal.
If you liked this, you may also like my 4-part series on The Future Analytics
Excellent post (and series), Jay. I like the idea of looking at it in terms of optimization - this brings us into the realm of looking at if, and if so what, ratios you are wanting to optimize. As Christensen discusses in the realm of disruptive innovations, you can improve on, for example, return on net assets by either increasing the numerator (e.g., increasing revenue which increases effort) or decreasing the denominator (e.g., decreasing assets such as people, machines, etc.). Either way, these are not specific to a big data scenario, but analogies can be drawn. For example, you could extend the scope of your analysis while at the same time reducing the spend (going from OLTP to Hadoop, for example).
From my point of view, the focus on operational efficiency goes exactly into the right direction. Coase's (1937) paper on "The boundaries of the firm" sparked the development of New Institutional Economics and essentially viewed transaction costs for, e.g., information retrieval, as the raison d'être for firms. In our big data world, we specifically reduce information costs in particular, and transaction costs in general, hence dissolving those boundaries - and we do that very much independent from the traditional, codified, "legal" boundaries along which firms are incorporated, as we dynamically create and modify vertical integrations along the services that we co-create. The real disruption, from my point of view, is not about those services moving incumbents upmarket, it is about completely new services being vertically created - very much on the fly.
Big data in that sense can be conceptualized around optimization, as you rightly do, and that optimization can serve well as benchmarks on the efficiency of big data endeavors - essentially, to which degree is any given project able to reduce transaction costs by better integrating and better focusing activities.
Big data can more broadly be seen as the only apparently mostly technical manifestation of a paradigm shift we see in all management sciences: The shift from telling to sharing (leadership), from invention to co-innovation (innovation), from inside-out to outside-in (strategy), from goods to service dominant logic (marketing), from financial to management reporting (finance) - and, ultimately, the minimization of transaction costs through vertical integration (operations).
This kind of consideration moves the focus away from the technical aspects and towards the actual business stakeholders and their benefits, allowing to create an analytical framework, based probably on understandable use cases, that each of the given group of stakeholders will be able to make sense of, and measure results against.
Thanks again for the excellent article!
M
Thanks, Matthias.
I like your comment on co-innovation, because I think this is crucial here. We need both the technology and domain knowledge for all of this. We need the technical skills to know how to put systems in place that support our solution, understand the various algorithms and predictive techniques and when to apply them and when not, think of ways to communicate any results through good data visualization practices, and tie that together with knowledge of the business process - how it works and best practices in the industry - as well as the wealth of experience of those running the day-to-day business.
We like to say "disrupt" a bit too much here in the Bay, when in most cases it is more a question of assist, support, collaborate, improve and optimize.
Very interesting point of view, I mostly agree, and I definitely agree with the general point that starting from a purely technology perspective does not work. At the same time, I do believe that some technology innovations we are seeing unfolding in front of us these days bring opportunities that can be unnoticed if we don’t look at our businesses with a specific perspective. When I interact with customers I don’t “look for Big Data opportunities” or “IoT opportunities”, but at the same time I try to go beyond business process optimisation, even if considered on technological steroids.
An approach I followed on Big Data, for example, is based on a simple provocation: if you, dear customer, had a crystal ball that could answer any question about your business, what would you do? Which question would you ask? Why? And what would you do with the answers?
It is an interesting game to play. I did it a few times, and in each and every case I ended up with a good list of ideas, or at least intuitions, of areas where a Big Data approach could actually make sense. And often enough the magic words “Big Data” are never even pronounced once!
It looks to me a kind of middle way (I do love middle ways!) between a purely technology drive, which often enough ends up with a big show and no business, and a purely business process approach, where definitely you can find so many opportunities, but not necessarily driven by the technologies and capabilities where we excel.
Thanks for your comment, Francesco! And I completely agree. This is what I was really getting at, but perhaps failed to show the other side of the coin. Yes, you still need to have the skills, understand the technologies involved, and know what's possible to help guide the conversation towards a solution, without just meandering about what could possibly be done if there were no constraints of any kind.
It is indeed somewhere in the middle. My focus has actually largely been on the technologies involved, to understand how they can be used once we know what problem we should tackle. In brief, then, this could be condensed to "you bring the use case, we bring the technology; you bring your domain expertise, we bring what we've seen elsewhere in your industry; together we'll achieve wonderful things".
It's also just a mental exercise to help come faster to concrete plans. I am convinced that Big Data and IoT will have an enormous impact on how business is conducted going forward. So when I say "Forget Big Data/Forget IoT", what I am actually saying is "these are a given, they will happen", not "it's not important".
Instead, rather to look for a "IoT" or "Big Data" use case, look for a use case that makes sense within your business operations. Once we have identified the problem area, let's go look for sources of data (or even devices and sensors) that can assist with that, both sources of data we already have (SAP Suite, EDW, etc.), as well as new data sources we haven't considered yet, or in the past might now have known how to deal with.