Big (Data) Does Not Always Mean Better

"Big data naturally appeals to many data geeks, high-priced consulting firms, and IT professionals" says Dr Mark Kotkin, former Director of Consumer Reports and now a market research consultant.

The essence of his recent article on Big Data is that data big or small, is only of value when it is subjected to robust analysis, but that poor data can never deliver results of worth.  What you may be including in your data analysis could have been open to bias and different data can produce quite differing stories.

Methodology matters.  So long as we keep in mind that any data set has strengths and weaknesses, the results achieved through Big Data analysis will be of value.

Real-time does not guarantee 'good times' for businesses without an understanding of context and some thoughtful analysis of results.

As Mark Kotkin reminds us "..while there is some general sense that big data arrives at warp-speed and involves huge data-sets from very diverse sources and methodologies, there is no consensus and little discussion of what comprises meaningful and valid “big” data-sets".

The age of Big Data also brings with it other hefty concerns; the security implications of having such data widely available, issues of privacy, ethical considerations of 'open' and 'social' data, and the impact on the political environment.

Being able to access petabyte and exabyte-size data-sets may have become increasingly commonplace but what are the broader implications for society?

Governmental Overview

Such considerations are exercising the minds of governments around the world including the White House, which has established a Big Data and Privacy Working Group.

They came to the following six conclusions:

1. Big Data saves lives - doctors can now monitor millions of data points providing more immediate and better diagnosis of a patient's condition.
2. Big Data makes the economy work better - transport, utilities and services save time and money and can adjust with alacrity to peak demand.
3. Bid Data saves taxpayer dollars - predictive analytics helps identify and counteract fraudulent activity
4. The balance of power between government and citizen is changed and can "chill the exercise of free speech or free association".
5. Intimate personal details can be extracted from Big Data sets - effective consumer privacy protection needs to be in place to counteract this.
6. Big data can lead to discrimination -  having our lives increasingly governed by algorithms and automated processes can potentially discriminate against certain societal groups, especially in the areas of housing, employment, and credit.

There is a growing realization that current privacy laws are outdated and social and private data already released globally, can never be retrieved.

But as with all data, big or small beware of over-hyping the subject and avoid spurious data associations and correlations.

Accept that Big Data can, and is, changing the world even though some can't yet see it. Obama’s re-election chief number-cruncher is on record as saying “Big data is bullshit” and a Cambridge professor summed it up in one word -  "Bollocks".

You may not know your 'Petabytes' from your 'Hadoops' but do you really need to?  Big Data is already making big decisions that affect our lives so we need to learn to live with it.

Related articles

• White House Takes Aim at 'Big Data' Discrimination
• Findings of the Big Data and Privacy Working Group Review
• The backlash against big data
• 5 Things to Lessen Your Anxiety About Big Data

As Strong As Your Weakest Link

As the old saying goes "You are only as strong as your weakest link".  Interestingly, in the recent rash of data disclosures from Edward Snowden and his predecessors, the weakest link could well turn out to be of America's own making.

Outsourcing key data activity to contractors rather than keeping it in-house means your online security is only as good as their employees are; in keeping mum about what they discover about your operations in the course of their daily duties.

Consider the fact, expounded by James Sensenbrenner in a recent editorial, that there are some 500,000 employees of private firms with access to the government's most sensitive secrets.  And this is just the States.  There are surely more in other countries contracted to undertake similar surveillance.

Some regard the actions of Manning, Assange and Snowden as heroic and others consider them heinous, but which ever side of the the ethical debate you sit on, the fact remains that confidential data was accessed and shared with those it wasn't intended for.

It is a sobering realisation (or maybe reconfirmation) that it is the low level IT guy who poses your greatest threat. These techs seem to be able to rummage through systems and make discoveries that evade all of the so-called safeguards that the governments throw at them.

Consider for a moment what you might have accessed online or sent to others via email in the past year.  I would suggest that many people who would feel less than comfortable in having a total disclosure of their online habits revealed to the world without their permission.

But is there anything you can do to mitigate the risk that others can and do spy on what you do?

Part of the answer could well have been given by NSA whistle-blower Snowden.  In reply to an online discussion set up by the Guardian newspaper he said that:

"Encryption (of email) works. Properly implemented strong crypto systems are one of the few things that you can rely on. Unfortunately, endpoint security is so terrifically weak that NSA can frequently find ways around it".

So there you go.  Even encrypting your email can only assure safe passage between systems and if the systems themselves have a weakness an IT tech on a mission can crack it or share it. Not the most comforting of thoughts and if the technician in question has a thumb-drive (as Snowden is reported to have by Senator Saxby Chambliss of Georgia), then your data could be shared and leaked well beyond the boundaries of your network.

Am I alone in thinking that these revelations could well have profound implications for the future of the Cloud? The 'contracting out' of data storage from your own servers to a third party based in another country could well have become a far more difficult decision for businesses to make.

And if you wish to mitigate some of the damage your email might cause you could always try using encryption yourself.  Here is one suggestion: GNU Privacy Guard for Windows which is free software.  Mind you I cannot guarantee that a low-level tech at the NSA hasn't already cracked it.

Related articles

• NSA Leaks Show Low-Level IT Worker as Secrecy's Weak Link - Bloomberg
• NSA-proof encryption exists. Why doesn't anyone use it?
• NSA leaker: No chance of fair trial
• Encrypting Your Email Works, Says NSA Whistleblower Edward Snowden
• Establishment Assertions that Snowden is a Chinese Spy are "Predictable Smears" (+video)