Microsoft wouldn't be surprised if Google is using some sort of custom-built mystery software that automatically shifts workloads between its mega data centers. After all, Microsoft is doing much the same thing.
"We are at such an enormous scale. Think about this world where many data centers and hundreds of thousands of servers are running search and enterprise services and all sorts of services," Microsoft data center chief Arne Josefsberg tells The Reg.
"These infrastructures that we run - and that Google does too - are so large, you can't really rely on individuals to manually make these decisions on an application failing-over from one [data center] site to another. Essentially, it all has to be built into automation software that makes these types of decisions."
This summer, at a cloud-happy mini-conference in San Francisco, Google architecture guru/quip-meister Vijay Gill hinted that the Mountain View Chocolate Factory had developed some sort of back-end technology that automatically moves live compute loads to other locations when a data center verges on overheating.
"You have to have integration with everything right from the chillers down all the way to the CPU," Gill said. "Sometimes, there's a temperature excursion, and you might want to do a quick load-shedding to prevent a temperature excursion because, hey, you have a data center with no chillers. You want to move some load off. You want to cut some CPUs and some of the processes in RAM."
And, yes, he indicated the company has a way of redistributing these workloads (near-)instantly. "How do you manage the system and optimize it on a global-level? That is the interesting part," he continued. "What we’ve got here [with Google] is massive - like hundreds of thousands of variable linear programming problems that need to run in quasi-real-time. When the temperature starts to excurse in a data center, you don’t have the luxury to sitting around for a half an hour...You have on the order of seconds."
Apparently, that bit about the "data center with no chillers" was a reference to Google's new facility in Saint-Ghislain, Belgium. According to a report from Data Center Knowledge, the Belgium facility really does operate without chillers, using nothing but the outside Belgium air - aka "free-cooling" - to keep temperatures low in the server room.
And it seems that when the Belgium summer gets too hot, Google uses its mystery software platform to shift loads elsewhere. Though the company won't actually fess up. "I don't believe we have published any papers regarding that," uber-Googler Matt Cutts recently told The Reg.
A typically coy Google remark? Microsoft seems to think so.