UK's Department for Work and Pensions continues to move off Oracle Enterprise Data Warehouse in pursuit of a single version of the truth
Redshift, Cloudera among preferred platforms, but after 7 years Big Red's system refuses to die
The UK's Department for Work and Pensions has confirmed it no longer runs a single data warehouse after moving analytics products off its Oracle system to a range of services for AWS, Oracle Cloud, Azure, and Google Cloud Platform.
This follows the £176.3bn-revenue department's award of a £3m contract to Methods Business and Digital Technology. The vendor has been charged with the migration of analytics products from Oracle Enterprise Data Warehouse (EDW) to the Data Services Platform, an AWS-based environment.
The contract award notice said: "A Re-engineering Discovery team will assess the current state of data in DWP warehouses and recommend an approach to migration/reengineer data to the Data Services Platform (DSP). Two Re-Engineering Delivery teams [will] design, develop end migrate required Products, redesigning manufacturing processes optimised for the new environments."
The DWP is an Oracle shop as far as its ERP and business applications go, although its long-term future with Big Red has been up in the air since the publication of its Shared Services Strategy in March.
But the hope of moving off the Oracle EDW dates back to 2014, following then digital transformation director general Kevin Cunnington's trip to sunny California, where he found "one item dominated the agenda: big data."
Presentations by Cloudera, HortonWorks, and others gave the civil servant "much food for thought as to how we could repurpose our existing DWP data warehouses away from Oracle, and toward a more open and distributed analytics architecture."
Cunnington, who went on to become boss of the gaffe-prone Government Digital Service before leaving to head the International Government Service, also happened to bump into none other than Palantir, the Trump-linked firm at the heart of a scandal over the NHS COVID-19 data store.
Needless to say, he "invited them to meet with Cath Hamp, director of DWP's departmental Security Design team," according to a government blog post.
But by 2017, the DWP was not quite ready to give up its Oracle EDW.
Instead, it elected to contract infrastructure company Redcentric [PDF] to re-host its data warehouse in a £3.8m deal, thinking that its in-house hosting system grew without appropriate controls. It opted for Oracle platform-as-a-service and database-as-a-service via a G-Cloud procurement.
In 2020, it was still supporting the system, although it was given the moniker Legacy Data Warehouse Infrastructure and set to be moved to the department's Remediated platforms, according to a £4m contract given to Deloitte for just eight months.
Meanwhile, the department continues to shift analytics and data warehouse jobs to AWS and its Redshift database among others.
A tender notice published last year is looking for a supplier to help with Management Information (MI) Enterprise Reporting noted that "multiple versions of the truth" is a problem data warehousing is supposed to solve. The current technologies used by the department's Data & Analytics team include React, Python, S3, Docker, Redshift, and AWS.
The department is understood to want to continue to use Oracle alongside other cloud-based data platforms such as AWS, Oracle Cloud, Microsoft Azure, and Google Cloud. It no longer has a single data warehouse and uses on-premises and cloud data systems.
Seven years after Cunningham wished to move the EDW off Oracle, the department might have to wait a while longer yet. ®