Building iRODS to take load off scientists' back

Panasas and WD/HGST go balls out for parallel file access and object storage integration


Panasas and Western Digital are combining high-speed data access and low-cost, bulk data storage for life sciences researchers through a partnership and integration exercise centred on the open-source data management software iRODS.

The use of the systems is described here. This is how it works:

  • Lab equipment generates data and open source iRODS (Integrated Rule-Oriented Data System) software gives it a thorough metadata going over, before moving the data from local workstation storage to an HGST Active Archive System, a disk-based object storage array optimised for high capacity and low cost storage with consequent low speed access. Amazon’s S3 protocol is used for the data movement.
  • An iRODS metadata catalog stores info about every file, every directory, and every storage resource in the iRODS data grid structure, and this aids data discovery. This grid structure federates zones, such as local workstation stores, the Active Archive System and a Panasas ActivStore array.
  • When life sciences researchers then need to process some or all of this disk archive-held data it is fetched, using iRODS rules and dumped into a Panasas ActiveStor array with its parallel file system access software.

  • Once the data is in the ActivStor them high-performance computing (HPC) servers access it in parallel and do their processing thing. In effect the Active Archive System is used to avoid storing all the data in the ActivStor, which would be unnaffordably expensive.

This multi-tiered, multi-product, system to enable fast access to a subset of a vast information store reminds us of Quantum’s StorNext, which can similarly use object storage (Lattus) as a backing store. Panasas competitor DDN has object storage (WOS) in its product set and this can be used as a backing store for its HPC arrays. See also this Fujitsu array and iRODS. ®


Other stories you might like

  • Talos names eight deadly sins in widely used industrial software
    Entire swaths of gear relies on vulnerability-laden Open Automation Software (OAS)

    A researcher at Cisco's Talos threat intelligence team found eight vulnerabilities in the Open Automation Software (OAS) platform that, if exploited, could enable a bad actor to access a device and run code on a targeted system.

    The OAS platform is widely used by a range of industrial enterprises, essentially facilitating the transfer of data within an IT environment between hardware and software and playing a central role in organizations' industrial Internet of Things (IIoT) efforts. It touches a range of devices, including PLCs and OPCs and IoT devices, as well as custom applications and APIs, databases and edge systems.

    Companies like Volvo, General Dynamics, JBT Aerotech and wind-turbine maker AES are among the users of the OAS platform.

    Continue reading
  • Despite global uncertainty, $500m hit doesn't rattle Nvidia execs
    CEO acknowledges impact of war, pandemic but says fundamentals ‘are really good’

    Nvidia is expecting a $500 million hit to its global datacenter and consumer business in the second quarter due to COVID lockdowns in China and Russia's invasion of Ukraine. Despite those and other macroeconomic concerns, executives are still optimistic about future prospects.

    "The full impact and duration of the war in Ukraine and COVID lockdowns in China is difficult to predict. However, the impact of our technology and our market opportunities remain unchanged," said Jensen Huang, Nvidia's CEO and co-founder, during the company's first-quarter earnings call.

    Those two statements might sound a little contradictory, including to some investors, particularly following the stock selloff yesterday after concerns over Russia and China prompted Nvidia to issue lower-than-expected guidance for second-quarter revenue.

    Continue reading
  • Another AI supercomputer from HPE: Champollion lands in France
    That's the second in a week following similar system in Munich also aimed at researchers

    HPE is lifting the lid on a new AI supercomputer – the second this week – aimed at building and training larger machine learning models to underpin research.

    Based at HPE's Center of Excellence in Grenoble, France, the new supercomputer is to be named Champollion after the French scholar who made advances in deciphering Egyptian hieroglyphs in the 19th century. It was built in partnership with Nvidia using AMD-based Apollo computer nodes fitted with Nvidia's A100 GPUs.

    Champollion brings together HPC and purpose-built AI technologies to train machine learning models at scale and unlock results faster, HPE said. HPE already provides HPC and AI resources from its Grenoble facilities for customers, and the broader research community to access, and said it plans to provide access to Champollion for scientists and engineers globally to accelerate testing of their AI models and research.

    Continue reading

Biting the hand that feeds IT © 1998–2022