Ice cold: How hard man of storage made Everest climb look easy

Rack 'em and stack 'em: The only thing in the cloud was Sagarmatha’s peak

Additional infrastructure elements

Previously, Gomes used Dell Precision and HP workstations. He currently uses some 30 Fujitsu Celsius workstations with hot-swap drives, Fujitsu outbidding Supermicro for the workstation contract.

The workstations run Linux, with Fedora and Centos, Gomes being a strong believer in open source and having only three Windows systems. “Most of our software tools are available for Linux, except Adobe Photoshop/ZBrush for which you need Windows or OS X,” he said.

Framestore is also a Linux house and WETA uses Ubuntu.

The network stack is basically HP with a Cisco firewall. An HPO c7000 BladeSystem is used for rendering in the data centre. It has 15 x dual socket hex-core Xeons in 10U with 36GB RAM per CPU.

Other possibilities

What about object storage? “I use files because the applications we use use files. Object storage is no good for our day-to-day workflow,” said Gomes.

And storing data in the cloud? He’s not really a believer in the public cloud, saying: “It’s just somebody else’s data centre [and] I don’t store data in the cloud because of latency and cost. We need real-time playback.” Real-time playback of 4K movies from the cloud is not possible and cloud storage is simply impractical.

How about using VDI instead of having workstations for each artist? “No. It’s better to have hardware under their desks for immediate response. With VDI I would feel I was offering a lesser service,” he added. Apparently some studios do use VDI and the artists get used to it and tune out the response lag involved.

Future technologies

Gomes is keen on the idea of parallel NFS. ”I’ve been dreaming about that for a while and am keeping my eye on NFS-Ganesha. It’s been a pipe dream for five years.” He doesn’t have the budget for a third-party pNFS product and is waiting for a usable open source version.

Parallel NFS would allow on-the-fly migration between his three storage nodes – a Holy Grail for Gomes, which he now has to manage manually. He is excited by the possibilities of bursting render compute to the cloud, using virtual FXT caches in Google’s cloud to cache his on-premises file storage to the cloud compute instances.

Following the Everest project RVX is now back down to about 20 digital artists, having staffed up for the film with freelancers.

By taking advantage of cloud compute bursting for some rendering work RVX could bid for projects involving more server resources than it has on-premises, staffing up once again with skilled freelance contractors.

It could also provide the flexibility needed if deadlines changed, with less time than previously planned available.

He is trying out vFXT in the Google cloud but there is a 49msec round trip latency to the nearest Google data centre in Belgium. “Basically, I wish Google had a data centre located nearer.” Even so, using vFXT and Google could be really good for heavy rendering calculations that takes days, and it "would fit one section of our workflow, if the studios agreed to its use with the potential security issues".

He’ll think about 3D TLC flash in the future with real-time playback of virtual reality files. But 3D XPoint is far away in the future, there being no need for it yet.

Gomes knows that VFX (video effects) data needs are growing. The Hobbit was filmed at 48fps in 4K. Stereoscopic left and right views for 3D double this. The next Everest will undoubtedly involve more storage and a heavier rendering workload.

Buoyed by the success of its Everest work, the RVX owners are also involved in creating a Reykjavik-based film and TV studio, similar to Peter Jackson’s New Line studios in New Zealand. Virtual Hollywoods can be based wherever there is a critical mass of digital artists’ expertise and VFX infrastructure.

And clever, fast caching front-ends like Avere’s FXT products can enable dense and cost-effective Infortrend and Supermicro commodity-based filers to be used for bulk storage, with the FXTs providing the instant response needed by the digital artists and their mini-HPC-style workloads. ®

Disclosure. Avere paid for my travel and accommodation costs.