How HPC should mean high performance storage, too

Big data shouldn’t mean slow answers

Sponsored When you’re asking big questions, you really don’t want to have to trade off accuracy and insight for speed.

In bioinformatics or pharmaceuticals research, the ability to sequence a genome in hours as opposed to weeks or months can dramatically accelerate innovation when it comes to spotting new disease variants or developing new vaccines or other treatments.

The development and training of self-driving vehicles generates vast amounts of data – in the region of 64TB per vehicle per day – which must be rapidly aggregated and prepped before it can be fed into the AI models underpinning autonomous driving technology.

And in commercial and financial organisations, there are multiple applications that rely on the analysis of vast amounts of data to enable real time, or near-real time, decisions such as high frequency trades, or customer recommendations in online retail.

In all these cases, traditional “big data” approaches based on commodity compute and storage architectures cannot deliver results quickly enough. This has led researchers and developers to look at what they can learn from the HPC world, where researchers have traditionally used highly parallelised architectures - and low-level languages - to crack compute-intensive problems such as physics simulations or climate modelling.

This is where High Performance Data Analysis (HPDA) comes in. Put simply, HDPA refers to the use of HPC techniques and platforms to accelerate data analytics workloads, meaning insights can be uncovered more quickly, even as the underlying data sets get ever larger. IDC data suggests that 67 per cent of HPC resources are now used for HPDA, with machine/deep learning and fraud detection applications the most common workloads.

Confronting the storage roadblock

Before we can fully enjoy the benefits of HPDA, however, we must first address the storage roadblock. It’s a well-known fact that throughout the evolution of computing, CPU performance has tended to increase much faster than that storage and I/O performance. Indeed, a data centre performance survey conducted in 2014 showed that CPU performance increased by 52 per cent, while memory and I/O performance increased by 6 per cent and 9 per cent, respectively, with storage performance showing the slowest improvement overall. This was not only down to the impact of physical media but also a result of the storage protocols.

So, why does this matter? If storage performance is lower than CPU and memory bandwidth performance, the data access capability will be lower than the data processing capability, which will hinder an organisation’s ability to deliver real-time services, reduce efficiency, and waste data centre resources.

The speed gap between computing, storage, and I/O becomes increasingly significant as computers evolve. However, the explosion of applications in emerging HPDA and HPC-based AI scenarios has highlighted the urgent need for real-time high-performance big data analytics, making it urgent we overcome the storage roadblock.

Pushing the boundaries of possibility

But, where there are challenges, there are opportunities. To balance the growth of computing and storage performance and power the HPC industry upgrade to HPDA, Huawei has developed its next-generation storage platform for HPDA, the OceanStor Pacific.

The lack of sufficient storage capacity can leave compute nodes underutilised, but still consuming power and IT budgets. With its ultra-high density design, Huawei OceanStor Pacific confronts this problem by increasing the capacity and performance densities per unit, minimising both footprint and TCO. Put simply, it gives you more, for less.

At the same time, the OceanStor Pacific is explicitly designed for the hybrid workloads typical in HDPA. One storage system supports both high bandwidth and IOPS, while each U provides 32 GB/s bandwidth and 400,000 IOPS, ensuring higher performance in all scenarios. Take oil and gas exploration, for example. Seismic data processing requires high bandwidth, while seismic data interpretation needs high IOPS. In this case, one OceanStor Pacific device can deliver on both goals, breaking the storage roadblock and making performance problems a thing of the past.

That’s not all – in emerging HPDA scenarios such as autonomous driving, precision medicine, and intelligent manufacturing, file, object, and HDFS services may be used in different phases of a pipeline. Traditionally, each of these three services would be provided by a separate storage device. This meant data had to be copied multiple times between devices, further impacting efficiency and wasting storage space. To tackle this problem, Huawei OceanStor Pacific supports lossless interworking of multiple protocols, including NFS, SMB, HDFS, and S3. This allows one data copy to be shared with multiple services. It’s smarter to keep things simple.

Unlocking the potential of data for something big

Effective data utilisation requires high-performance storage and computing capabilities. Without this, data is simply clutter in the virtual world. That’s exactly why Huawei launched the OceanStor Pacific next-generation storage for HPDA; through the integration of cutting-edge technologies, it eliminates the gap between computing and storage performance and breaks through the storage roadblock. This helped it earn recognition from judges at Interop Tokyo 2021, where it won Best of Show Award Grand Prize in the Server & Storage Division. Thanks to the product's unique technical advantages, such as multi-protocol interworking, hybrid workloads-oriented, and high-density design, it delivers high efficiency and outstanding performance.

As a result, the OceanStor Pacific has been widely adopted in HPC scenarios across various industries. Whether it’s powering ground-breaking scientific research, improving meteorological data analysis efficiency, or enhancing oil and gas data processing, Huawei OceanStor Pacific is helping organisations to overcome the storage roadblock, turn data into discoveries and fully embrace digital transformation.

For more information about Huawei’s OceanStor Pacific Storage for HPDA, please click here.

Sponsored by Huawei