Exascale kit and vendors
SKA is also lucky because it’s quite a sexy project for vendors to be involved in. Aside from the science-benefitting-mankind bit, SKA will be the biggest data-led project in the world - when it gets off the ground.
“I’ve been to many exascale meetings in the last few years and the SKA has always been singled out as having massive data flows and massive compute and it’s that combination that is quite unique. I believe that’s why vendors of computers and cloud services are so interested in us, because we’re pushing both of those envelopes at the same time,” Cornwell said.
Although the team has their big budget to build the specific software and algorithms they need for their data, they’ll also be making use of existing software like Hadoop and Swift to help them handle their mountains of raw information. Cornwell is especially keen on getting some of that data and computing into the cloud.
“In the long term, you can see that astronomers don’t really want to be running data centres and so sometime between now and 20 years will be a transition point to the cloud,” he reckons.
Cornwell has been talking to Amazon about the possibility of using its AWS for data storage and computing, as well as helping the team get the scientific discoveries out to other astrophysicists and astronomers.
“In the near future, we expect to announce a grants programme with Amazon and that will facilitate use of the cloud by astronomers around the world to get hands on experience,” Cornwell said.
Yet even with something like AWS, the full SKA will still dump zettabytes of raw data per year that it simply can’t afford to store. With an operations budget of 60m euro a year divided between people, power and storage what can be retained will be governed by raw money.
“We will have to throw away all the raw data, all the stuff that we sent to the supercomputer, after we go through a lot of processing and averaging the data to produce images and those images are the things that we would keep. Even in some cases, we can’t afford to keep the whole image, we have to keep just little parts of it,” he said.
That’s the situation for now, but Cornwell is confident that will change in the future. He still believes in Moore’s law for compute power and he reckons bandwidth will expand too.
“We re-invest on a five-, ten-year timescale. We end up putting new digital signal processing in, more bandwidth and then that normally brings you a huge new set of capabilities. One of which might be the ability to image the piece of the sky you’re looking at every millisecond. We can’t afford to do that at the moment, but it is quite conceivable that in ten years time, we could.”
Even without the boost in power, the SKA is set to make huge strides in astronomy. ®