The puzzling misnomer about so called ‘big data’ is that it is not that big. It may contain data in formats that IT people have traditionally found hard to handle but in terms of data volumes the amounts involved are small by comparison with the volumes that have been used by supermarkets for more than 15 years already.
The second rather puzzling aspect is that Big Data attracts so much attention. The mechanisms for processing and analysing such types of data, albeit in smaller volumes, have been used by market researchers for at least 30 years. What has changed is the volumes of such data and their mechanism of collection.
Of much greater interest, as well as greater challenge, is real time analytics.
Techniques developed by economists in the late 1970’s based on the application of control theory to economic and social data suffered from the omission of attitudinal data.
Data sources which now paint a continual picture of consumer behaviour, consumer circumstances and consumer attitudes are about to render such techniques all powerful.
The application of control methods and feedback loops has moved on unrecognisably since the 1970’s. We see this manifest itself in every defence industry application used around the world. And in the car you drive every day.
Rapid response to changing conditions is essential to the effective management of the these systems. These techniques are being easily adapted to enable one-to-one response to changing customer needs and situations.
When you book a ticket you spark a whole series of interactions that will eventually take us into the realms of the “Minority Report” era.
There is, however, still much work to be done before we get there. The IT guys still struggle to discern mathematically tractable measures from analogue data like tweets and photos that captures things are predictive rather than just descriptive.
But the process of solving this is well under way and it will be completed within 5 years if not sooner.
RedRoute’s customer panel technology, for example, provides effective and systematised methods of tracking and predicting the impacts of changing consumer sentiments.
Our Effective Net Preference measure adds the dimension of consumer attitudes that was missing from earlier attempts to understand and predict the trends in consumer needs and behaviours. Our clients already benefit from this everyday to help them outperform the market.
Going forward these techniques will become ever-more automated and continuous. Leveraging real time feedback and analytics to enable businesses to respond to changing consumer needs.
Predicting the future is always hazardous. The problem is that ‘we don’t know what we don’t know”.
We can, however, react quickly when it changes and that is the benefit that control theory and feedback loop analytics provides.
Will it be a substitute for creative genius? No, never. The human mind is still light years beyond the power of computers to ever be truly creative.
In the meantime, however, these techniques will enable us to at least react more quickly to better anticipate and meet consumer needs.