Skip to Content
Business Trends
Author's profile photo Werner Daehn

Big Data and Analytics in the Example of Retail

The retail market is an interesting area to look at, lots of changes and opportunities. And certainly one with large volumes of data, assuming the Big Data trend is about only. One important aspect for any company is understanding the buying behavior and influencing it. But is that new? Certainly not. Initially the promise of Data Warehousing had been to let users find out such new information, then Data Mining was supposed to find that out automatically. Now Predictive Analysis and Big Data algorithms are used. Same goal, different marketing terms? Not really. There are more chances but some gotchas still.

The beer and diapers story

I am sure everyone has heard the beer and diapers story, where when analyzing the data, a significant correlation between the two product was found and exploited. Ignoring the details of this story, let’s simply assume it to be true and consider how this information could be exploited.1429373819_2e1493fd18_b

  • Place the two products nearby: Everybody buying either beer or diapers will see the other product and be reminded about the other. Revenue of those two does increase.
  • Place the products in opposite corners: Customers are buying both products, both are on their shopping list. Placing both far away means the customer will browse through the entire store and hopefully find more he needs. More revenue overall.
  • Increase the price of one and decrease the other: Make sure customers know about the cheap price of diapers and as they will buy the beer anyhow, put a premium on that. In sum more profit is made.
  • Bundle both: Either a real product bundle or advertise both together to get more subconscious attraction – this shop seems to provides me with what I need.

The point is, having new insight and using it, are totally different challenges. Or in other words, a BI or Big Data strategy without the means of executing on the results is a waste of time and energy.

Captain, there is a temporal anomaly!

A few weeks ago I bough a large SD card for my camera at Amazon. Checking my recommendations, Amazon advertised a large Compact Flash memory card to me, because of that buy. Although the CF card has an attractive price, top of the edge ratings and performance I am pretty sure I will not fall for it. I do not have any device that can use a Compact Flash card anymore. So the recommendation was a total failure, for obvious reasons. It is hard to tell how the algorithm came to the conclusion I might be interested but just looking at the data, it seems like a good guess. I have bought CF cards in the past. I buy memory cards regularly for various devices. I am sure there were quite a few customers who bought both.

Issue is, CF card sales had been big in the past but is close to zero today. The recommendation algorithm does not take the time aspect in a correct perspective.

Machine learning

Data Mining and its Big Data successor of machine learning is trying to help here. It knows the customer, the product, the time of the purchase. Through feedback loops the system learns which relationships are important, which customers are of similar kind and at the end this model can, provided with new input data, produce an output based on what it has learned. In my example it might get as input that I bought a lot of cameras and related equipment over they years and therefore categorizes me into the customer bucket of photographer. I recently bought a new camera but everybody else in that bucket bought a new memory card as well. There is some likelihood I might be interested in such. This approach might look at the face of it a lot like the “Customers who bought this also bought..” recommendation logic, but with machine learning the algorithms are more fault tolerant. Pure statistical methods to find a set of similar customers with my buying profile and building a list of items they bought which I did not, will yield to the same result, yes. But only if there is a significant match. I bought Nikon cameras, somebody else bought Canon cameras. Are we the same or different? It depends, we both might be interested in the same high speed memory cards but I am pretty sure the lenses we have for the cameras will not be the same products. The more the customer is constraint into the exact same profile – bought exactly that item – the more unique the customer will be. So machine learning and its fault tolerant algorithms are supposed to provide the better result, if used properly and with care.

You don’t know what you don’t know

Looking at my company revenue data it is obvious sales is declining in one region. We need to take immediate action to prevent further damage, say invest in marketing? Lower our prices? Excellent response, realtime action and maybe the proper answer. But maybe not. If the root cause is that the stock levels are at zero in that region, say Venezuela these days, it would be the worst thing you can do.

In such an obvious example we laugh about it but all too often we can see such situations in our own life. The problem is the “you don’t know what you don’t know” paradox. When building a BI system you try to limit the effects via two approaches. First by providing easy access to cross references, e.g. if the stock level amount is highlighted when viewing the sales numbers because both are available at a similar level (product, shop, time), it might make people think. And second by an easy way to distribute findings for further validation by others. In the big data space this problem gets even worse as the algorithms do not have an understanding about the world. Way too often they simply output what the user thought it will, because of unintentionally maneuvering the tool into that direction.

Another less extreme example: For one particular cake we can clearly see a repetitive pattern. For a few days the revenue is high, then it is trickling and suddenly it is high again. Seems to be a three week period but not strictly. The stock is a bit weird, when there is a fresh delivery the revenue kicks in again but the slow revenue is when there is 40% stock level still. Any idea?

The products shelf life is months, so it could not be the freshness of the product. Any better idea? Looking at the shop in person revealed the miracle. This cake is sold in two variants but with the same UPC/EAN code. The chocolate version has a high demand and makes up half of each palette. The citrus version of the cake is less popular. Once all the chocolate cake was sold out…. What is the root cause? The vendor selling both versions as one to cut costs.

Let me predict

The goal of a Data Warehouse or a Analytics project was essentially to dwell in the past. The Point of Sales data was typically uploaded at the end of the day, aggregated and then made available for comparative analysis for the next day. Nothing wrong with that, the far majority of decisions being made have a long term impact. And a lot can be learned from the history (although when looking at the politicians today you might not feel like that). The problem is a different one, everybody is doing that, hence no competitive advantage. With data being more in realtime the reaction time can be cut and for some areas it allows to get another percent better. If it is raining outside, putting the umbrellas into a more visible spot might drive customers to buy one. If the stock level declines rapidly to zero, you will not ask for a delivery truck – there are physical limits. The main point in this example is it is a reactive approach. First the event, then a reaction. If there is the option to predict an event and its consequences, it would allow for more options. So you are trading the uncertainty of a prediction against more flexibility. In this example, the stock is usually allowed to drop to 10% but because of the weather prediction and its impact on sales, the minimum stock is temporarily 20%. Probably not worth it for this example but a visual example at least.

The main point I am trying to make is that success in the market comes hand in hand with predictions. That is the daily business of the managers at various levels. Should I invest in an area because its economy is thriving? Should I de-invest because of political instability? Is there a demand for a product? These decisions cannot be automated, they are way too complex and need an understanding about the inner workings of the world. But if a short term prediction on a single customer can be made, that is a different story. Do not wait until the customer bought something but try to predict his interest. That is where Big Data can help with its capability to process large amounts of data for rather simple predictions. Now the system would suggest me the brand new high speed memory card which I don’t need but as it would allow me to utilize my camera even more, I might be tempted. If the prediction is wrong, so what. But if it is correct…

Win-win situations

Most of the above is about the retail company taking advantage over the customer. Trying to influence him. This can be a winning proposition but ultimately will fail, it is just a matter of time. I would argue the success of Amazon is less the result of its market power but the value it provides to me as a customer.

  • One place for all: I do not have to register in one shop for electronics, another shop for cloths, … one login for all products.
  • Delivery: I have a good feel for the delivery times. If I need something immediately and are willing to pay a premium, I drive to the next local dealer. Else I rather look for a reasonable price and can wait a day or two.
  • Customer service: If I buy from Amazon directly and not its Marketplace, I have never been disappointed.
  • Price comparisons: Due to the marketplace I have a rough feel where the price stands and can balance pros and cons. For expensive articles I use specialized web sites but quite often buy from Amazon at the end – cheapest overall or not so much more expensive than others.

Where is this win-win feeling when going to the next supermarket? They are all the cheapest – according to the advertisements at least, they all have similar goods, they all allow me to get the products I need easily. This personalization is what is missing. I would think that Big Data and predictive algorithms can help here as well to generate a win-win situation.Cash_Registers

Imagine I would create the shopping list at home by scanning or typing the products I need. Ooops, last carton of milk? Scan it with the smart phone, put three on the shopping list. Soap was not bought in ages, the app suggests it unobtrusively. At the shop, this list is reordered automatically based on the shop floor layout. I pick the first product, scan it with the phone and it is put off the list. At the same time, the app knows where I am in the shop as I just scanned the product. So it can direct me to the next. If on my way there is something else I might be interested in, it can highlight that in a pleasant manner. At the end I upload my entire list of bought items and pass though the self-scanning cashier. Next time I update my shopping list at home, these additional bought items do show up as well in case I want to buy them again. Based on my buying habits I can see what others bought and try out their products by putting them at my shopping list.


Point is, be creative. There is new technology available, its costs are reasonable low, they can be tried out easily thanks to cloud solutions. But technology at the end has to serve a single goal, to increase the business numbers. Either by cutting costs, by increasing sales directly, by increasing sales via customer satisfaction and best, by doing all of the above.


Assigned Tags

      Be the first to leave a comment
      You must be Logged on to comment or reply to a post.