OVH punts hybrid water and immersion cooling for high performance systems
Air by itself just doesn't cut it anymore
European cloud operator OVHcloud has just celebrated 20 years of using water cooling technology in its datacenters, leading up to the development of a hybrid system that combines both water and immersion cooling to drive greater efficiency when operating demanding workloads.
OVH, like some of the larger cloud operators, has been building its own server hardware for many years, and the advantage of this is that the company has been able to adapt them to any kind of cooling system.
At this point we have to mention, before our readers do, that there is no connection here with fire suppression systems and the like. Regular readers will know that two of the company's facilities in Strasbourg went up in flames in 2021, although it has since rebuilt a datacenter on the site with walls rated to withstand fire for two hours, with fire-rated shipping containers housing batteries outside the datacenter.
When it comes to the company's use of liquid cooling, in the meantime, it was initially driven purely by a desire to pack more servers into a rack to increase density in the datacenter and thereby lower costs, according to the company's Global Environmental Director, Grégory Lebourg. However, it has also been beneficial for environmental factors, such as cutting energy and water use, he claimed.
"Most of the technology used today and back then is about air cooling by a circulation of air refreshed by a Computer Room Air Conditioning (CRAC) unit, and those units are very inefficient because air does not capture the heat well, plus that type of equipment to cool down the aisle is very inefficient in terms of power use," Lebourg told us.
For this reason, OVH looked at capturing the heat at its source, putting a water block on top of the CPU (and later GPU), through which water circulates to absorb the heat, then it is extracted outside of the rack and ultimately outside of the building.
But not all the heat is generated by the CPU, some of it comes from other components such as the memory, hard disks or SSDs, so OVH still uses air cooling here, with the airflow through the server to an air-water heat exchanger at the back of the rack, which it calls the "fridge door," with fans to pull the air out of the of the rack.
"The beauty of this system is instead of having many fans per server to create the airflow, we need fewer fans, because most of the heat is captured by the water blocks, and fewer fans means less power," Lebourg claimed.
Another benefit is that the rack is fully autonomous, Lebourg said, meaning self-contained, so there is no need for a raised floor or walls to separate hot and cold aisles. This allows for higher density, such that OVH claims that its datacenters are smaller in size than rival operators for a given number of servers.
Because of this cooling system, OVH does not need to control the inside air temperature or the humidity level, so it can use existing buildings rather than purpose-built datacenter structures, according to Lebourg.
"For example, in Quebec, where we started our first international expansion back in 2012, that used to be an old aluminium factory from Rio Tinto," he said. "On the west coast in Oregon, the datacenter used to be a Piano Factory, in Virginia it used to be CIA facility, in Germany, it used to be a print factory and in London, it used to be a British Telecom computing center," he said.
Thanks to this system, OVH claims that its datacenters have a power usage effectiveness rating (PUE), a measure of energy efficiency, of 1.28 that compares with an industry average of 1.55, according to figures from the Uptime Institute.
But this is just the current situation, and OVH has now developed a hybrid cooling system that is intended to offer the benefits of water block cooling and immersion cooling by effectively combining the two.
The company developed this approach because it was not satisfied with the immersion cooling technologies that were already on the market, Lebourg said, finding them compromised in one way or another.
"One way to do immersion cooling is to put server blades in a dielectric fluid, and have a pump circulate it around those servers. This is efficient, but you still need energy to run the pumps," he told us.
An alternative is to ditch the pumps and use natural convection via a heat exchanger inside the tub. But the downside is that fewer servers can be fitted, possibly half the number, according to Lebourg. So while it is efficient in terms of power, it is an inefficient use of space.
A third approach is to use a two-phase system, in which the dielectric fluid is converted into vapor by the heat, creating a natural convection between the liquid and the gas.
"You don't need pumps anymore, and you can still put 48 servers inside the tank, but this is very costly and tricky to handle because you need to have a sealed cover because of the vapor," Lebourg explained.
Instead, OVH looked at adapting its own existing cooling technology.
"One thing we do good is to capture the heat at the CPU level with a water block. And most of the heat is coming from there, so it's very efficient in terms of PUE. So why not keep that and use immersive cooling for the parts which are today cooled via airflow?" Lebourg asked.
This means keeping the water block for the CPU and GPU while using immersion cooling in dielectric fluid for the rest of the components, which the company likes to claim as offering greater efficiency.
- It's time to stop fearing CPU power management
- The cloud can reduce greenhouse emissions, but don't assume it's automatic
- Why Microsoft is really abandoning evaporative coolers at its Phoenix DCs
- French cloud operator OVHcloud gets datacenter funds from EU lending arm
"What we've tested in our lab is that the partial PUE, which is just the measurement of energy you use for the cooling itself, stands at 1.004,” Lebourg said. "So you only need a very limited amount of energy to cool down your system, because you create a natural convection from the water block and you do not need a pump, and there is still the benefit of the overall water cooling system developed by OVH in terms of transporting the heat outside of the building."
OVH has apparently filed for 30 patents covering various aspects of this hybrid cooling, and so far has just a single pilot deployment comprising a single rack at its experimental datacenter near Lille in Northern France.
The design requires a change in the rack architecture used by OVH to one where the servers are mounted vertically instead of stacked horizontally. But instead of having one big tub for an entire rack of server blades, OVH has opted for one tank per server to make it easier in terms of operations.
However, despite coming up with this new, more efficient liquid cooling system, OVH intends to use it only for infrastructure in its cloud that is built for demanding workloads, such as AI, gaming or high performance compute (HPC) applications. Unlike some others in the industry, it sees no need to immersion cool everything, at least not yet.
"This will be for very high density racks, let's say between 44 to 88kW," said Lebourg. "Those types of requirements are mainly driven by the gaming industry or artificial intelligence. So the way we look at it, it's a niche market today, but we are ready to go that route, and we are ready to deploy that."
This view contrasts with that of Cisco, which last year told our sister site The Next Platform that even mainstream servers are soon approaching the point where air cooling will not be enough, and liquid cooling will be necessary. Immersion cooling will likely be the norm, at least for greenfield deployments, the company said.
Lebourg said he did not see this happening in the near future, although OVH should be able to swiftly adapt as it builds between 50,000 and 70,000 of its own servers each year.
"Most of what we build today is still leading to racks which are consuming between 8kW and 12-14kW. We do have some HPC racks which are going up to 40kW, but this is not the majority," he said.
"So most of what we are manufacturing today can still be cooled down by air cooling solutions, which is inefficient in terms of power use, but it works. Or it can be cooled down by our technology easily. So I don't see the rush."
However, liquid cooling and immersion cooling have been rising up the agenda lately, especially at last year's Supercomputing Conference where myriad different solutions were on show.
Other providers have also been getting in on the act, with Northern Europe datacenter operator GlobalConnect claiming earlier this year to be the first colocation provider in Europe to offer immersion-based cooling to its customers, and saying it expects to expand the service across all of its datacenters in future. ®