In many organisations the costs of data warehousing are expressed on the balance sheet as the hardware, software and administration costs that support the operation.
In practice, many hidden costs are associated with the data warehouse with the structure of an organisation.
So says Mark Thomas, IBM’s Infosphere and data warehousing specialist. “The cost of a complex data warehouse is more than that expressed on the balance sheet,” he says.
“It is more than the hardware, the software and the people that work directly with the data warehouse. There are business units outside the IT organisation that almost certainly have people working with the data warehouse as well.”
In, out, shake it all about
These staff may be used to get data in to and out of the data warehouse for analytics or reporting. Thomas cites a recent project with a large bank where it was discovered that 106 database administration staff were associated with the data warehouse in IT and across lines of business.
“These were staff who were accessing and manipulating the data – not productive work but getting data in and out, and keeping the surrounding business intelligence environment going,” he says.
“There’s a huge amount of cost involved in keeping data warehouses going, just in moving data as well as indexing and partitioning and tuning. There can be a lot of people involved just to make sure the data is being jammed in and the associated processing work goes on.
“It’s a very inefficient use of manpower, just keeping infrastructure working rather than creating new insight or driving business analytics for competitive advantage.”
Companies are used to living with this state of affairs. Thomas says many of them recognise the scale of the costs and inefficiencies associated with the data warehouse, but because these costs tend to be apportioned around the organisation there is often little impetus to do anything about it.
He cites the case of another bank where the department that was the main user of the data warehouse was accessing it for free. No one had thought to charge the unit for it, and the department managers certainly weren’t going to bring this oversight to anyone’s attention.
Thomas adds that often the internal cost charges associated with moving data in or out of the data warehouse can be prohibitive, preventing analysts from trying new things.
IBM’s response to this can be summed up in two somewhat ungainly words: “pre-integration” and “pre-configuration”.
With IBM’s big data suite an awful lot of the costs usually associated with data warehousing are removed, the company claims. The new breed of appliances championed by IBM includes technologies from Netezza, the data warehousing appliance company.
The vendor claims the result is a highly flexible and robust warehouse which merges the traditional and big data approaches, coping with both structured and unstructured data. It hugely simplifies the database environment because all the components are optimised to work together.
“Resource can be used to get projects out of the door”
“You have a far more manageable environment and scarce IT resources can be better used to create value,” says Thomas.
“It’s a simple and manageable data warehouse which changes the cost paradigm from a staffing perspective. Resource can be used in a more productive manner – to meet business requirements and get projects out of the door, rather than just keeping the machine going.”
Scarce resources are deployed more effectively and implementation costs are also much lower, IBM claims.
Change the tuning
“So much comes pre-integrated and pre-configured,” says Thomas. “Very often implementations are about getting data into the right applications. That remains, but there is no configuration so implementation is very simple and quick.
“We have a client who has had a Netezza appliance for about a year, and as yet hasn’t needed any training. It is all so simple and it can’t be altered that much.”
Thomas concludes that a traditional data warehouse is a hugely costly beast because it requires an infrastructure that is too complex, requires too many people to maintain it and too much tuning to meet the ever changing needs of the business, particularly at a time when IT is under ever greater pressure to reduce costs.
“Netezza and our associated information management tools offer massive performance out of the box, without the traditional need for tuning, indexing or partitioning,” he says.
“It doesn’t require a team of expensive specialists and significantly speeds up time to market of the deliverables from the data warehouse.” ®