The parable of the beer and diapers

Never let the facts get in the way of a good story


Database myths and legends (Part 3) BI (Business Intelligence) is about extracting information from data and data mining is an important part of that process. Data mining is a process that looks for patterns in data, so in a sense it is like querying the data. The crucial differences between simply querying the data and data mining can be summed up as intent and scale.

When humans query data we start with an idea, such as: "I think that we sell more DVDs to males than to females." And then we run a query to test the idea and the answer either confirms or disproves our hypothesis. A data mining algorithm doesn't have ideas. It has no intention of testing ideas for the simple reason that it doesn't have any.

What it does have is huge processing power at its disposal, so it simply tests a very large number of possible correlations. This can be done by firing a very large number of queries at the database, but that approach can be very slow.

The algorithms used nowadays often use more sophisticated approaches; they can, for example, create a multi-dimensional data structure and then examine it looking for patterns, and/or outliers. When they find something of interest, they flag it for attention.

It is easier to illustrate the difference between querying and data mining with a good example and, already firmly enshrined in BI mythology, is the "beer and diapers" story.

It goes (with minor variations) like this:

Some time ago, Wal-Mart decided to combine the data from its loyalty card system with that from its point of sale systems. The former provided Wal-Mart with demographic data about its customers, the latter told it where, when and what those customers bought. Once combined, the data was mined extensively and many correlations appeared. Some of these were obvious; people who buy gin are also likely to buy tonic. They often also buy lemons. However, one correlation stood out like a sore thumb because it was so unexpected.

On Friday afternoons, young American males who buy diapers (nappies) also have a predisposition to buy beer. No one had predicted that result, so no one would ever have even asked the question in the first place. Hence, this is an excellent example of the difference between data mining and querying.

The story goes on that, once the correlation was uncovered, it was easy to back extrapolate from the effect to the cause.

  • Young American males frequently indulge in ritualised carousing behaviour with friends of Friday nights.
  • Carousing usually involves the consumption of beer.
  • Most young American males only buy diapers after they have fathered offspring.
  • Offspring acquisition is a known carousing inhibitor.

So the proud new father is walking around the store on Friday afternoon. He knows there is no way that he is going to get out of the house to join his mates at the bar. However, there is nothing to stop him from drinking beer at home. All he needs is to be reminded of that fact. After seeing the results of the data mining, Wal-Mart moved the beer next to the diapers and beer sales went up.

Like all good myths, the beer and diapers story does have its origins in fact; but sadly, almost all of the detail (and specifically the detail that makes it a great BI story) is probably fabrication. We are all indebted to Daniel Power for uncovering the origins of the story. He provides all of the detail here.

In short, he traced the story back much further than I would have believed, way back to 1992. At that time, Thomas Blischok was the manager of a group at Teradata. His group looked at point-of-sale data from Osco Drug stores - 1.2 million baskets worth in all. It isn't clear what tools the team was using. They are described as "state-of-the-art, query generation tools", which were undoubtedly leading edge in 1992.

Those queries revealed that, between 5pm and 7pm, customers tended to co-purchase beer and diapers. No correlation with age or gender was established; although it isn't clear whether these questions were ever asked.

In addition, the store chain does not appear to have exploited the information by moving the products around. So we have a correlation between beer, diapers and time, but no correlations with age, gender or day. Worst of all, it is very much open to question whether the techniques used would qualify as data mining in its modern sense.

So, where does all that leave our beer and diapers story as an example of data mining? Well, I for one am prepared to accept that it didn't really happen in the way my grandfather described it to me as I sat upon his knee. Sigh.

However, that doesn't mean that we have to consign the story to Room 101 (for those readers outside the UK, also a popular TV comedy). The story is so popular because it is a good illustration of the difference between querying and data mining. The facts don't change that at all.

The image of a forlorn young man trudging round a supermarket on Friday night; his day suddenly brightened by the sight of a stack of beer sitting incongruously next to a pile of diapers is somehow wonderfully compelling. And none of us would have asked that question in the first place.

There is no reason we cannot continue to use it as an illustrative story; as long as we are aware that it is simply an allegory or fable. In fact, given that it is a fable about a set of data, we could call it a table fable. ®


Other stories you might like

  • Running Windows 10? Microsoft is preparing to fire up the update engines

    Winter Windows Is Coming

    It's coming. Microsoft is preparing to start shoveling the latest version of Windows 10 down the throats of refuseniks still clinging to older incarnations.

    The Windows Update team gave the heads-up through its Twitter orifice last week. Windows 10 2004 was already on its last gasp, have had support terminated in December. 20H2, on the other hand, should be good to go until May this year.

    Continue reading
  • Throw away your Ethernet cables* because MediaTek says Wi-Fi 7 will replace them

    *Don't do this

    MediaTek claims to have given the world's first live demo of Wi-Fi 7, and said that the upcoming wireless technology will be able to challenge wired Ethernet for high-bandwidth applications, once available.

    The fabless Taiwanese chip firm said it is currently showcasing two Wi-Fi 7 demos to key customers and industry collaborators, in order to demonstrate the technology's super-fast speeds and low latency transmission.

    Based on the IEEE 802.11be standard, the draft version of which was published last year, Wi-Fi 7 is expected to provide speeds several times faster than Wi-Fi 6 kit, offering connections of at least 30Gbps and possibly up to 40Gbps.

    Continue reading
  • Windows box won't boot? SystemRescue 9 may help

    An ISO image you can burn or drop onto a USB key

    The latest version of an old friend of the jobbing support bod has delivered a new kernel to help with fixing Microsoft's finest.

    It used to be called the System Rescue CD, but who uses CDs any more? Enter SystemRescue, an ISO image that you can burn, or just drop onto your Ventoy USB key, and which may help you to fix a borked Windows box. Or a borked Linux box, come to that.

    SystemRescue 9 includes Linux kernel 5.15 and a minimal Xfce 4.16 desktop (which isn't loaded by default). There is a modest selection of GUI tools: Firefox, VNC and RDP clients and servers, and various connectivity tools – SSH, FTP, IRC. There's also some security-related stuff such as Yubikey setup, KeePass, token management, and so on. The main course is a bunch of the usual Linux tools for partitioning, formatting, copying, and imaging disks. You can check SMART status, mount LVM volumes, rsync files, and other handy stuff.

    Continue reading

Biting the hand that feeds IT © 1998–2022