Our world has everything connected. Data is readily available. Humans spend their time producing digital breadcrumbs. And all this data can be collected and analysed in ways that are new and never before imagined.
When discussing big data sometimes we worry about the difficulty of managing data being produced. The difficulty lays because the data is heterogeneous, because it appears in high volumes and with variety. This led to the appearance of the ‘paragon’ 3V (Variety, Velocity, and Volume of data).
But this corresponds to two views of our society. One that is technical. Another is social.
The growth in data, the growth in the capacity to analyse it, and the growth of Echelon to gather communications in real time are inevitable. These are infrastructural problems. The view of the world as a technological system. A system that wants to extract more and more information, process it in a sequence of marginal gains to find and tell (usually the verb is sell) us the secret patterns we didn’t know about ourselves.
The socio-cultural factors show a picture of society changing. People produce more data because people want commodity. The commodity to press a button in the fridge and place automatic orders for milk on the local store of choice. Were cars will be autonomous and safer because they talk to each other. Where people want to share their photos with family. And in all these processes some (or many) metadata will be shared. The Internet of Things allows anyone to convert a simple electric toy with a microcontroller to repurpose it and connect it to the IoT. We are all happily contributing to the digitisation of everything.
The data produced has always been a problem for those that want to understand it. For them knowledge is paramount. They can be monks copying ancient manuscripts in the medieval ages being technological challenged when wanting to copy an entire library. Or they can be IBM deploying Apache-Spark as the next big hammer after Hadoop. There is always a limitation to the amount of data society can process. And there is always some limitation to the amount of information that one can extract from that data. There is always a limit, and although the term Big Data is new, the problem isn’t.
The digitisation of the world we know is a social phenomenon that is growing with each new device sold. The life simplification of digitisation causes the production of streams of data and metadata that some consider excessive. Security can be a problem, but changes will happen, not because technology imposed by big companies, but because society needs the changes to accommodate our new needs. We want to be connected and functional in the modern world.
This discussion leads me to think that we are dealing with two views of the Big Data. One, that treats Big Data as the technological challenge of gathering, processing, and acting upon the world. Where Data is increasingly being produced and promoted as fueling advances of the future. And for this, we’ll need to be able to understand those streams of data. Another is to look at this as a sociological phenomenon where the growing digitisation of the planet is producing changes at a faster pace. In it Big Data is just a side effect of this social change. So, is Big Data Cause or Consequence?
This seems to be a chicken and egg problem. Is Big Data the analysis of more and more data, and with that analysis the cause of new knowledge that is transformative to the society? Or is the digitisation of the world and the social changes observed, producing a drastic change in the output of their digital signatures, that Big Data is a side product of?
The truth is that the two phenomena might be interconnected in a positive feedback mechanism. The more we see behavioural changes in the population, the more data will be produced and the more the technicalities of Big Data will be publicised. The more Big Data guardians have knowledge about the world, the more will they act on it, asking for more data. If you imagine what society can be if you read an encyclopaedia, imagine what it can produce when Watson receives enough information to decide the next fashion trend.
The present is the inventor of the future. And the question comes down to individual participatory choices (at least in democratic worlds). You cannot escape the future, but you can choose how much you want to engage with it. Progress (as a transformative force) is upon us every day. But progress is a society concept, not a technical one. And how much you engage is always going to be relative to a moving baseline, and not to a fixed reference point. The digitisation of the world is making Earth a better place (even if will be too hot to live in it soon).
Today’s Big data challenges are just the same of the past. How society changes and technology evolves are like the two faces of a Möbius strip. Both interconnected, always affecting each other in eternal motion. Always on an eternal continuum like present and future.
