Oh no, you're thinking, yet another cookie pop-up. Well, sorry, it's the law. We measure how many people read us, and ensure you see relevant ads, by storing cookies on your device. If you're cool with that, hit “Accept all Cookies”. For more info and to customize your settings, hit “Customize Settings”.

Review and manage your consent

Here's an overview of our use of cookies, similar technologies and how to manage them. You can also change your choices at any time, by hitting the “Your Consent Options” link on the site's footer.

Manage Cookie Preferences
  • These cookies are strictly necessary so that you can navigate the site as normal and use all features. Without these cookies we cannot provide you with the service that you expect.

  • These cookies are used to make advertising messages more relevant to you. They perform functions like preventing the same ad from continuously reappearing, ensuring that ads are properly displayed for advertisers, and in some cases selecting advertisements that are based on your interests.

  • These cookies collect information in aggregate form to help us understand how our websites are being used. They allow us to count visits and traffic sources so that we can measure and improve the performance of our sites. If people say no to these cookies, we do not know how many people have visited and we cannot monitor performance.

See also our Cookie policy and Privacy policy.

This article is more than 1 year old

Lustre HPC file systems no longer disappear in a puff on Amazon Web Services – but you'll pay more

Throughput boosted and encryption in transit added to temporary counterparts

AWS has added persistent file systems for its FSx Lustre storage service, which lets you use a high-performance clustered file system on demand.

Lustre (from "Linux Cluster") is a distributed file system designed for high performance and widely used in supercomputers. The AWS FSx for Lustre service was first announced in late 2018 as a scratch file system, the idea being that you copy data from the AWS Simple Storage Service (S3) to temporary Lustre storage, process it there, and then copy the results back to S3. The specification was 200 MB/s throughput per TB of storage provisioned, and IOPS (input/output operations per second) of "millions".

AWS has now said it will include enhancements including a second-generation scratch file system that can burst to 1,200MB/s throughput per TB, and encryption in transit to add to the existing encryption at rest, subject to some conditions about instance type and region.

Creating a Lustre file system in the AWS console

Creating a Lustre file system in the AWS console

The biggest news is that persistent Lustre file systems are now supported, with "automatic data replication and file server failover". Performance is not quite as good as with the second-generation scratch storage, but you still get IOPS of "millions" and options for throughput of 50, 100 or 200MB/s per TB provisioned. The minimum provisioned amount is 1.2TB or "increments of 2.4TB", according to the docs. Pricing is per GB/month starting at $0.164 for the slowest option (in London, other regions vary), which is about $200 per month for the minimum 1.2TB. It is more expensive than scratch storage, which has a single $0.162 GB/month for the highest performance.

In order to use FSx for Lustre, you will need a Linux VM with the Lustre client, or you can use integration with EKS (Elastic Kubernetes Service) or SageMaker, the AWS model training service. Use cases suggested by AWS include geospatial analysis, seismic processing, financial modelling, media rendering, electronic design automation, big data analytics and machine learning.

A strong feature of FSx for Lustre is that you can create a file system via a simple form in the console, or through the FSx API. Both Microsoft Azure and Google Cloud Platform also have Lustre, but you need to deploy your own cluster. Azure has templates for Lustre file systems here and similarly GCP describes how to create a Lustre file system cluster here. ®

 

Similar topics

Similar topics

Similar topics

TIP US OFF

Send us news


Other stories you might like