Too much data - how to decide - Part 1.

Brain Overload ?

There is so much information available now - some would say too much and you would think that management decisions would have improved from this surplus of information. 

However it seems that it has not improved and we need to see what we need to do to improve that step as well.

The information below outlines the challenge and in step 2, some solutions.

Tony Park - Business Gardener.

Part 1

Management Decisions in a Data-Rich Environment

October 15, 2013 in

When making a decision, today's business executive has access to roughly a thousand times more data, and a thousand times more powerful analytic and computational tools, than would have been available for making the same exact decision as recently as 1993. Unfortunately, however, the quality of the management decision-making process itself has not much improved.

Today’s managers are woefully under-prepared for using data and evidence to support intelligent, scientifically sound decisions, and most business executives are completely unaware of just how flawed their reasoning processes really are.

Let me give you an example. When I address business audiences I sometimes pose a simple question based on a case study. A few years ago, a European book club found that by calling its new members within their first month of membership, just to welcome them on board and inquire whether they fully understood the terms, first-year members bought 8% more books, and their likelihood of renewing their membership at the end of the first year increased by 6%, as well. No selling at all was allowed on these welcome calls; the calls were made purely to improve the customer experience.

Are you with me so far? OK, now I ask the audience:

How can the book club possibly know it was the calling program that achieved these results, all by itself?

Couldn’t it have been the selection of books being offered that year, or the arrival of a particularly successful bestseller? Or what about a fall-off in competition from some other book club? Or even an improvement in the country’s overall economy, for that matter? And what did the calling program achieve in the following year, anyway?

I have asked this very simple question of business executives in audiences all over the world, and all I usually get are perplexed looks. The vast majority of managers are simply flummoxed by the question. Only about half the time, in fact, does even a single executive suggest the right answer. And it is a very simple, very basic answer.

The book club knows that it had to be the calling program all by itself that achieved these great results because they didn’t call every new member. They only called a random sample of new members that year, and then they compared the purchase volume and retention rate between those who were called and those who were not. It was really that simple.

This kind of “A/B test” has become quite common in the age of interactive marketing, because now the vast majority of marketing activity is based on individual interactions with individual customers, the way the book club's business model worked years ago. If you want to know what offer to make to a particular kind of Web visitor, or what graphics will generate the most click-throughs, then randomly select a portion of your Web visitors to receive treatment A, and another portion to receive B. Google, Amazon, and other large, online businesses are capable of conducting and evaluating thousands of individual A/B tests daily.

A/B testing, however, is just one aspect of dealing more scientifically with data. And in future posts I’m going to talk about a number of additional issues, including:

• Knowing when you can trust data, and when you can’t
• How to avoid the rationality-impeding influences of your own human biases
• Testing the null hypothesis
• Planning ahead for randomness and unpredictability
• Using the Law of Large Numbers and other principles to improve your decisions

If you think you’re already pretty skilled with the kind of reasoning required to make better management decisions in a data-rich environment, then you should have no problem answering this follow-up question:

Once the book club conducted its test and found that the calling program did, in fact, have positive results, they rolled the program out and began making welcome calls to all new members. What should they expect now?

1. The percentage improvements will be the same?
2. The improvements will not be quite as good?
3. The improvements will be even better?

And why should they expect this?

To get the answer to this question, you can click here.

More in part 2.
Tony Park