Big Data and analytics: Reg survey crunches the numbers

Dazzling new solutions or irritating new hype?

Every IT professional understands that relational databases play an important role in most organisations. Indeed, the previous article in this series highlighted that such repositories are used to hold business critical data in many organisations. Such “traditional platforms” are not only widely deployed, they are also well understood, something that cannot be said for a number of more recent additions to information management armouries (Figure 1).

Indeed the chart above indicates that on average almost as many respondents have no familiarity with modern solutions such as scale-out storage, distributed search and indexing, WORM (Write Once, Read Many) databases, distributed data analytics and rules based stream processing as have a reasonable knowledge of them. Such results come from a web survey of a community likely to have a higher than normal number of respondents interested in ‘big data’ solutions. Experience therefore leads us to believe that the levels of understanding and working knowledge of these modern solutions is almost certainly even lower in the IT population as a whole.

Given the likely low levels of familiarity, just how widely deployed are these solutions, many of which could play significant roles in big data systems?

The technology of analytics and data storage

Taking into account the results of Figure 1, it comes as little surprise that relational databases and other “Legacy” platforms are very widely utilised. Specialist high performance RDBMS configurations are also well used. Amongst the other options only OLAP multi-dimensional databases and scale out storage platforms achieve significant levels of use in around one in five organisations. Beyond these, in line with the relatively low levels of understanding discussed above, other technologies and approaches have yet to make a major impact (Figure 2).

However it is already evident that much greater use of such systems is forecast by significant numbers of respondents. With such a welcome potentially close at hand, it will be interesting to see how rapidly the vendors of such solutions can educate broader sections of the general IT community to raise awareness and comfort levels.

There is another challenge hidden here that has been gnawing away for many years, one which big data and other infrastructure developments of the past few years are now throwing into the spotlight. This concerns the licensing models and costs associated with using database systems in modern environments (Figure 3).

The chart highlights that only one in ten respondents are of the opinion that database vendors are ready to provide licensing models suited to their needs, especially as business requirements become ever more demanding. Nearly half give a categorical ‘No’ and disagree with the statement. Almost one in seven are unsure.

A couple of freeform comments from respondents in relation to licensing add a little colour:

“Whenever you are talking RDBMS, and management is fixed on avoiding open source solutions, you can count on the solution to be very expensive. It is also shocking to see all of the extras that sneak in over time and cost even more!”

And this comment really sums things up:

“While many vendors are talking the talk, when it comes to really performant enterprise solutions, they are not proactively offering us useful alternatives”

Even though the database vendor community is not alone in attracting such unhappiness in its licensing models, the despair evident here makes this a challenge that software vendors should ideally address in the short term. However, it’s evident that some are dragging their feet as they enjoy additional income from the mismatch between traditional commercial models and new deployment options.

So with understanding and use of “big data” solutions at relatively low levels, this raises the question of how pressing is the demand to implement such systems to analyse big data sources such as those emerging from high volume data streams?

High volume data feeds

In the emerging realm of big data, much is made of the potential for organisations to extract nuggets of valuable information from very large, unstructured data sources. One example of this envisages organisations analysing high volume data feeds such as event logs, click streams or security logs etc. in something approaching real time. But is there much demand to analyse such data pools? The answer is a qualified ‘yes’ (Figure 4).

As can be seen from Figure 4, around one in five respondents report they already make extensive use of high volume data feeds from either internal or external sources, with over a quarter using such data sources in at least some areas of the business. Only around one in four states they do not make use of such feeds and don’t expect this to change.

But such numbers need to be considered in context of the web survey, whereby experience tells us it is likely that such a survey on big data is more likely to attract those already engaged, or at least interested, in these areas. So whilst we can state that high volume data feeds are already an important source of data for some organisations and are likely to grow, they have yet to become massive.

Thus quotes such as the ones below relating to examples of high volume external information sources that are important to our survey respondents need to be considered in context.

“We collect some state and initialization data from bespoke electronic systems, as well as integrating non-critical feeds from other end-user systems such as Twitter and Facebook, all in Realtime”.

We use .. “web clicks, data from Twitter, Facebook and other social media”

We record … “Web logs, click streams, impression logs, Data center usage data, to the per-core or per-VM level, CDR, IPDR, xDR, Financial quote streams / logs”

With clear potential for growth, just how well are vendors doing positioning their big data offerings and are they perceived to be ready to support deployment in businesses at large?

Big Data – understanding, vendor support and solution positioning

Given the lack of familiarity of new technology solutions, we asked a few questions to put things into the ‘Big Data’ context. To sum things up, it appears that a significant number of respondents believe that technology is now ready to help them address existing problems and tackle new challenges (Figure 5).

However, when it comes to how well the survey’s respondents think vendors will be able to help them exploit these new solutions, things are not so positive. (Figure 6)

The most important matter thrown up by the chart shown above is how few participants in the survey agree with the premise that they have a clear idea of the business benefits that big data could deliver or have a good idea of what technologies are becoming available to them. Clearly these are challenges that the vendor community needs to address urgently if the extensive marketing around big data is to translate into organisations taking such systems on board in anything other than niche cases or in test scenarios.

But just how well vendors will be able to achieve this in the short term must be questioned when the chart also shows that only one in ten of those surveyed feels convinced that vendors and consulting firms are in a position to provide potential customers with the support and services they will require to get big data solutions running to the benefit of the business.

This is a monumental challenge that vendors and consultants alike will need to address very quickly, preferably as a community rather than as individuals if potential customers are to be won over. It is also an issue that must be addressed before organisations will believe that big data is for everyone, not just for a few organisations close to large suppliers who can drop in a fully trained data scientist or two.

This leads on to the key question, just what do readers of the Register think about big data. Is it a solution ready to fly or a gale force marketing wind blowing hype through the industry?

Big Bang or Burst Bubble?

The IT industry is very familiar with extended marketing campaigns around the next great revolution that will change the face of the industry for ever and a day. Within just the past few years we have had SOA, Cloud, Private Cloud, Social Media and Bring Your Own Device, to name but a few. Is ‘Big Data’ seen to be the next subject of the hype cycle (Figure 7)?

Alas, the answer to that question is a resounding yes. Irrespective of the potential merits in the underlying technologies, over three quarters of those taking part in the survey consider the term big data to be over-hyped and the associated marketing around it to be unhelpful. Only four per cent disagree, a number remarkably small when many of the survey respondents work in IT services and consulting organisations.

This also needs to be seen in the context that a minority of respondents in Figure 6 state they have a clear idea of the advanced data storage and big data analytic technologies that are now being brought to market.

If big data is to deliver the benefits that new technologies are making available there are a number of factors that the vendor / supplier / consultancy communities need to address. Chief amongst these is the need to educate people more on what big data solutions can actually deliver in the way of benefits as well as raising the awareness of just where such solutions might, and might not, fit.

But beyond this, there is a danger that the huge cloud of smoke being thrown up by some over-indulgent marketing could seriously dampen big data’s credibility as something for organisations of all shapes and sizes, not just something for the chosen few. At the very least, a perception of marketing hype could certainly delay adoption of solutions and slow down the realisation of the benefits it could bring to many organisations.


This article follows on from an earlier piece where the results of a survey of Reg readers was used to illustrate the core state of business information and analytics in organisations today, including a review of which data types are most important and which are growing fastest. The survey was run during August and September of 2012 by Freeform Dynamics with participation from 502 readers of the Register (for sample breakdown see here).

Similar topics

Similar topics

Similar topics


Send us news

Other stories you might like