The term “cold storage” has something of the morgue about it. Dead bodies in refrigerated cabinets, what an image. Facebook is backing the use of a photo morgue, the use of cold storage for old pix.
Old photos will have low individual access rates but when access is wanted it is wanted at faster speeds than a tape library could produce, as tape cartridges have to be robotically collected from their slot and transported to a drive, before being streamed to find the right photo.
The Facebook Open Compute Project has produced a cold storage specification (pdf) which states: "a cold storage server [is] a high capacity, low cost system for storing data that is accessed very infrequently … is designed as a bulk load fast archive."
What's the best technology for it? According to the Open Compute Project (OCP):
The typical use case is a series of sequential writes, but random reads. A Shingled Magnetic Recording* (SMR) HDD with spin-down capability is the most suitable and cost-effective technology for cold storage.
This needs its own infrastructure, namely a cold storage data centre, and the document provides a schematic of one such data centre, holding 744 racks. These cold storage racks are taller (39 Open U) and heavier than standard Open Racks.
Facebook's OCP cold storage schematic diagram
There are 16 Open Vault storage units per rack. Each unit is a 2U chassis with 30 x 3.5inch SATA 3Gbit/s hot-plug disk drives inside it; not 6Gbit/s SATA "due to signal integrity concerns for SATA 6G."
OCP Storage Vault
Let's do some arithmetic:
- A rack has 16 storage vaults, meaning 16 x 30 drives = 480 drives
- A row has 24 racks: 24 x 480 = 11,520 drives
- The data centre has 30 disk rack rows: 30 x 11,520 = 345,600 drives
- Assuming they use 4TB SMR drives: 4 x 345,600 = 1,382,400TB
That's 1.38 exabytes of raw capacity. Wow!
From the OCP, we learn: "To achieve low cost and high capacity, Shingled Magnetic Recording (SMR) hard disk drives are used in the cold storage system. This kind of HDD is extremely sensitive to vibration; so only 1 drive of the 15 on an Open Vault tray is able to spin at a given time."
Also: "In the typical use case, only one HDD is powered on at a time in each HDD tray, so at most only two HDDs will be powered on in a 2xOpenU Open Vault system at a given time."
Preventing vibration damage to the SMR drives seems crucial:
Because SMR HDDs are used in the cold storage system, in addition to the requirement for only spinning up one HDD in a tray for reading/writing, a special spin-up control mechanism is needed during the system power on process to avoid the vibration from/to each other. Not all the HDDs spin up at first power on.
The system has a disk spin-down function as a power-saving measure:
To be aligned with power saving strategy in future data centre operations, the Open Vault design also supports a spin-down control. Both the hardware design and enclosure management firmware implementation support such a feature.
The cold storage spec states:
A HDD spin controller ensures that there is only one active disk in a tray.
• Power-on a specific HDD before accessing it. It may need to spin-up if has been recently spun-down.
• Power-off a specific HDD after it finishes access.
• Within an Open Vault system, power-on the two HDDs in the same slot on both trays.
This cold storage vault design is specific for an archive of data (photos) with relatively fast access needed. We imagine other cloud application service providers might have similar needs. Indeed, Taiwan-type ODMs (Original Design Manufacturers) could also benefit as mainstream storage array vendors wouldn't be able to ship such arrays, given their current cost structures and business models.
Facebook's picture morgue could be the death of mainstream array vendors' cold storage supply hopes. ®
Read up on shingled magnetic recording here. Seagate has shingled drive products coming.
El Reg first wrote about Facebook's cold storage data centres here.