How big is the big data business? That depends on how you dice and slice it.
The box-counters at IDC have cooked up a very precise and formal definition of big data that will keep everything from being thrown into the big data pot. To some ways of thinking, big data is just the new ERP for a webby world, and to others it is just the next logical extension of the back-office systems that have been created for the past four decades on systems of various kinds.
IDC says there are three kinds of metrics for big data systems it is using in its market analysis.
- The first is that the system has to collect over 100TB of data, and the word "collect" is precise in that in many cases companies are storing data in memory and processing it within that memory without pushing it out to disk, as happens in online transaction processing and data warehousing systems. Big data is also about data in motion, not just data at rest (whether on disk or in main memory), and IDC lumps in systems created by companies to capture and process real-time, streaming data, such as systems used in conjunction with trading systems for raw data and analysis to drive trades or smart metering systems for power distribution networks.
- IDC is also not requiring for data sets to be large to be called big data, but does require for those data sets to be growing at a rate of 60 per cent or higher per year to be considered big data. Big data is not just about the data set, but the explosive nature of the data set.
- Finally, IDC requires for big data systems to be deployed on "scale-out architectures," which means clusters of servers and storage, and that big data systems be wrestling with data in at least two different formats.
When you look at the servers, storage, networks, software, and services that comprise this big data market as defined by IDC, you get a market that was worth $3.2bn in 2010 and that is expected to grow at a compound annual growth rate of 39.4 per cent over five years to hit $16.9bn by 2015. That growth rate, says IDC, is seven times higher than that of the overall IT market over the same five-year span.
"While software and services make up the bulk of the market opportunity through 2015, infrastructure technology for big data deployments is expected to grow slightly faster at 44 per cent CAGR," said Benjamin Woo, program vice president for storage systems at IDC, in a statement accompanying the big data market analysis. "Storage, in particular, shows the strongest growth opportunity, growing at 61.4 per cent CAGR through 2015. The significant growth rate in revenue is underscored by the large number of new open source projects that drive infrastructure investments."
IDC estimates that the server portion of the big data market is growing a little more slowly than the market at large, with 27.3 per cent growth compounded over the five years, compared to 34.2 per cent for software and 61.4 per cent for storage.
The market researcher believes further that there will be an increase in the use of big data appliances in the coming years as well as more customers outsourcing their big data jobs or relying on cloudy infrastructure from third parties to chew on their monster data. The shortage of experts in various big data software technologies and in those who know how to do analysis on the data once it has been chewed will encourage vendors to create cloud-based services that do the chewing on behalf of customers. No matter how jumpy some customers might be about the security of data that leaves their firewalls, if there is a run on big data talent, they may have no choice but to jump to a cloudy data munching service if they want to stay within a budget. ®