Big Data has crossed the chasm from hype to everyday reality remarkably quickly. Its adoption has been accelerated by hungry data warehousers using big data techniques to get better answers from their data mining activities.
Attending the TM Big Data InFocus 2014 conference in Amsterdam, it became abundantly clear that hype and promise-laden claims by vendors of big data software and storage were simply not there. The language and the approach has changed about face in just a couple of years.
The surface cliches are still there (“From Information to Actionable Insight”, for example) but they are realistic for the big data miners attending the conference.
The presentations mainly focussed on retail and telco suppliers, and there were also sessions on big data and financial services. These are three areas where there are millions of customer records and a great need to maximise overall customer spend by financial, retail and telco service suppliers.
Strange but true
What seemed odd, at first, was that big data and Hadoop were not the first words on everyone's minds. It was just there, a third leg added to the data warehousing and business intelligence stool. The concepts of getting intelligence from data could have been borrowed wholesale from a DW/BI conference five years ago.
There's nothing wrong with that. The data miners basically want the same thing now as they wanted then; the ability to comb through masses of data and get actionable insights from it.
There was much talk of widening the sources of the data, including social media for example.
We learnt that big data techniques could be applied to small data collections, such as vaults of just a few terabytes in size, as well as massive, petabyte-scale vaults. Big data is more about data analytics techniques than big (meaning massive) data collections.
This has stopped some storage vendors in their tracks, for they have not sold masses of storage capacity for the masses of Big Data-hitting customers. The amount of stored Big Data seems to be less than anticipated.
Old dog, old tricks? Not so much
It is also about as getting as close to real-time as possible. A presenter even mentioned how big data analytics could be used to warn financial call centre operators about possible fraud as a call transaction was taking place. That was leading edge stuff, however, with much big data analysis taking hours or substantial fractions of hours in comparison.
The tone of the presentations was measured and realistic about how Big Data analytics are being used. Service vendors like AWS and Splunk focussed on customer use cases. Telcos talked about reducing churn. Retail operators talked about the loyalty card data. It was all so familiar and resonant of Data Warehousing and Business Intelligence concerns, although some technologies were new - like transcribing call centre conversations into text and analysing them.
One aim of that was to check if operators were misunderstanding things customers said about products, or times and dates.
Big Data is a business intelligence concern, about service and retail suppliers predominantly, although health figured somewhat.
Free from hype – and glamour
The organisers, TM Forum, is a not-for-profit organisation. Vendors can sponsor events but such sponsorship doesn't come with automatic speaking slots. Speakers have their pitches reviewed and product selling is not permitted.
This big data event is a hype-free zone and there is little interest in the fortunes of vendors, and the sexiness of coming technologies. The tone tends to realism and earnestness, meeting the concerns of practitioners, not raising hype about new products.
This was the second TM Big Data event and the organisers expected it to be less-well attended than the 2013 event because of that. In the event it was just as well attended, over-subscribed in fact, with 150 attendees and overflow seating in the presentation hall.
To the attendees looking to learn from others about how to better accomplish their goals, it was satisfying. They learnt more about how to solve their business problems than about technology. To this hack, expecting Big Data vendor fireworks it was a surprise.
Big Data analytics are here now, hype-free, and in everyday use by data miners avidly looking for better tools to source, collect, store, filter and analyse data. It's real, it's crossed the chasm and we glamour-hunters can move on. ®