Section 9.1 Big Data in Context
MarketWatch (a Wall Street Journal Service) recently published an article with the title, “Big Data Equals Big Business Opportunity Say Global IT and Business Professionals,” and the subtitle, “70 Percent of Organizations Now Considering, Planning or Running Big Data Projects According to New Global Survey.” The technology news has been full of similar articles for several years. Given the number of such articles it is hard to resist the idea that “big data” represents some kind of revolution that has turned the whole world of information and technology topsy-turvy. But is this really true? Does “big data” change everything?
Business analyst Doug Laney suggested that three characteristics make "big data" different from what came before: volume, velocity, and variety. Volume refers to the sheer amount of data. Velocity focuses on how quickly data arrives as well as how quickly those data become "stale." Finally, Variety reflects the fact that there may be many different kinds of data. Together, these three characteristics are often referred to as the "three Vs" model of big data. Note, however, that even before the dawn of the computer age we’ve had a variety of data, some of which arrives quite quickly, and that can add up to quite a lot of total storage over time (think, for example, of the large variety and volume of data that has arrived annually at Library of Congress since the 1800s!). So it is difficult to tell, just based on someone saying that they have a high volume, high velocity, and high variety data problem, that big data is fundamentally a brand new thing.
With that said, there are certainly many changes afoot that make data problems qualitatively different today as compared with a few years ago. Let’s list a few things which are pretty accurate:
-
The decline in the price of sensors (like barcode readers) and other technology over recent decades has made it cheaper and easier to collect a lot more data.
-
Similarly, the declining cost of storage has made it practical to keep lots of data hanging around, regardless of its quality or usefulness.
-
Many people’s attitudes about privacy seem to have accommodated the use of Facebook and other platforms where we reveal lots of information about ourselves.
-
Researchers have made significant advances in the “machine learning” algorithms that form the basis of many data mining techniques.
-
When a data set gets to a certain size (into the range of thousands of rows), conventional tests of statistical significance are meaningless, because even the most tiny and trivial results (or effect sizes, as statisticians call them) are statistically significant.
Keeping these points in mind, there are also a number of things that have not changed throughout the years:
-
Garbage in, garbage out: The usefulness of data depends heavily upon how carefully and well it was collected. After data were collected, the quality depends upon how much attention was paid to suitable pre-processing: data cleaning and data screening.
-
Bigger equals weirder: If you are looking for anomalies - rare events that break the rules - then larger is better. Low frequency events often do not appear until a data collection goes on for a long time and/or encompasses a large enough group of instances to gran one of the bizarre cases.
-
Linking adds potential: Standalone datasets are inherently limited by whatever variables are available. But if those data can be linked to some other data, all of a sudden new vistas may open up. No guarantees, but the more you can connect records here to other records over there, the more potential findings you have.
Items on both of the lists above are considered pretty commonplace and uncontroversial. Taken together, however, they do shed some light on the question of how important “big data” might be. We have had lots of historical success using conventional statistics to examine modestly sized (i.e., 1000 rows or less) datasets for statistical regularities. Everyone’s favorite basic statistic, the Student’s t-test, is essential a test for differences in the central tendency of two groups. If the data contain regularities such that one group is notably different from another group, a t-test shows it to be so.
