The essence of his recent article on Big Data is that data big or small, is only of value when it is subjected to robust analysis, but that poor data can never deliver results of worth. What you may be including in your data analysis could have been open to bias and different data can produce quite differing stories.
Methodology matters. So long as we keep in mind that any data set has strengths and weaknesses, the results achieved through Big Data analysis will be of value.
Real-time does not guarantee 'good times' for businesses without an understanding of context and some thoughtful analysis of results.
As Mark Kotkin reminds us "..while there is some general sense that big data arrives at warp-speed and involves huge data-sets from very diverse sources and methodologies, there is no consensus and little discussion of what comprises meaningful and valid “big” data-sets".
The age of Big Data also brings with it other hefty concerns; the security implications of having such data widely available, issues of privacy, ethical considerations of 'open' and 'social' data, and the impact on the political environment.
Being able to access petabyte and exabyte-size data-sets may have become increasingly commonplace but what are the broader implications for society?
Governmental Overview
Such considerations are exercising the minds of governments around the world including the White House, which has established a Big Data and Privacy Working Group.They came to the following six conclusions:
- Big Data saves lives - doctors can now monitor millions of data points providing more immediate and better diagnosis of a patient's condition.
- Big Data makes the economy work better - transport, utilities and services save time and money and can adjust with alacrity to peak demand.
- Bid Data saves taxpayer dollars - predictive analytics helps identify and counteract fraudulent activity
- The balance of power between government and citizen is changed and can "chill the exercise of free speech or free association".
- Intimate personal details can be extracted from Big Data sets - effective consumer privacy protection needs to be in place to counteract this.
- Big data can lead to discrimination - having our lives increasingly governed by algorithms and automated processes can potentially discriminate against certain societal groups, especially in the areas of housing, employment, and credit.
There is a growing realization that current privacy laws are outdated and social and private data already released globally, can never be retrieved.
But as with all data, big or small beware of over-hyping the subject and avoid spurious data associations and correlations.
Accept that Big Data can, and is, changing the world even though some can't yet see it. Obama’s re-election chief number-cruncher is on record as saying “Big data is bullshit” and a Cambridge professor summed it up in one word - "Bollocks".
You may not know your 'Petabytes' from your 'Hadoops' but do you really need to? Big Data is already making big decisions that affect our lives so we need to learn to live with it.
