Everybody is talking about how life will never be the same again as we enter the Age of Big Data. Here, for example, is David Brooks' new column on Big Data. As I've mentioned before, I started working in Big Data 31 years ago. After I got my MBA in 1982, I joined a newish firm that was the first to effectively exploit for marketing research purposes the inundation of data from checkout scanners at supermarkets and drug stores.
We paid to put the new laser scanners in every supermarket and drug store in four small cities, such as Eau Claire, Wisconsin. We recruited 2,500 households in each of our four test markets to identify themselves as they went through the checkout lane, so we could track every single consumer packaged goods purchase they made. We could finally answer quantitatively an endless number of questions that brand managers and marketing professors had dreamed up since WWII about consumer behavior. Do people who buy Tab by the six-pack also buy Lean Cuisine when it's on sale? Any question like that imaginable, we could answer.
Furthermore, we could do over 30 years ago what Jim Manzi called for in his recent book Uncontrolled: use these towns and giant real world laboratories, where we could control the TV commercials seen by each of our panelists in their own living room. If P&G wanted to test a new Mr. Whipple spot for Charmin, we could divide our panel up into two cells with identical Charmin purchasing over the last year, then show one cell the new commercial and one cell the old commercial, then record which cell bought more Charmin over the next year.
We caused a sensation in the consumer packaged goods world. Wall Street said the world would never be the same and our stock doubled on the day it went public in March 1983.
For awhile, we made a lot of money, so it definitely changed our world.
After awhile, though, brand managers at P&G got tired of paying hundreds of thousands to find out much how sales would go if P&G management granted their fondest wish of having their advertising budget doubled. The usual answer was: not much, if at all. The only time doubling an already generous P&G-sized ad budget was likely to move the needle was if P&G had some real new news about their brand to convey (which, being P&G and investing heavily in chemists to improve their products, they actually sometimes did).
As this lesson sank in, it set off a mini-recession in TV advertising around 1986. So, did testing. Every year after that, we'd write into our budget that this would be the year that the world would rediscover how incredible this testing service was, but instead sales just eked slowly away. The company finally shut down the service late last year.
By 1987, however, we'd moved into even bigger Big Data, collecting all the supermarket sales for about 25 million people.
That too changed the world. Yet, somehow, the world kept spinning on its axis.
Let me offer another, more well-known example: baseball. I first became fascinated by baseball statistics in 1965 when I was six. Let's see if I can remember Ken Boyer's 1964 MVP line: .293, 24 homers, 111 RBIs. ... Nope, it was really .295 / 24 / 119.
A vast amount has been written since then about how Big Data has revolutionized baseball. For example, today we know that Ken Boyer wasn't really the best player in the National League in 1964. No, a vast amount of statistical analysis has uncovered the electrifying news that the best player in 1964 was actually — and you'll be stunned to learn this — Willie Mays!
Oh, wait, everybody back then knew Willie Mays was the best player in the National League. I was six and I knew. According to the latest Big Data analysis, Willie had been the best player in the league for 9 of the previous 11 years, just like most six-year-olds would have more or less guessed.
So, how much has baseball changed since 1965 due to the famous revolution in Big Data? Michael Brendan Dougherty says we are in a Golden Age of baseball, while Ross Douthat has caveats, citing my post on moneyball making baseball worse.
To check how much has changed, I turned on the radio to listen to the Dodgers like I did in 1965. Some things have changed, but others haven't, such as Vin Scully, who was a veteran announcer in 1965 with 16 years experience with the Dodgers, is still doing the play-by-play in 2013.