The universe in a cloud

The MeerKAT telescope, the first phase of the Square Kilometre Array (SKA) telescope in South Africa, will allow astronomers to understand how galaxies form and evolve, study dark matter, black holes and understand some of the most violent explosions taking place in the universe. The trouble is that before we can unlock these secrets of the universe, we have to solve the accompanying big data challenge.

“The MeerKAT is an amazing instrument with fantastic science goals,” says Professor Russ Taylor, director of the Inter-university Institute for Data-Intensive Astronomy (IDIA) and SKA research chair at UCT and the University of the Western Cape (UWC). “The catch is that the answer to these and other questions about our universe lives in a massive stream of data.” Unless scientists can develop the tools to work with data on that scale, the universe will remain a mystery.

Speaking at the launch of the IDIA research cloud at the Iziko Planetarium and Digital Dome on Monday, Taylor said that the massive data sets to be produced by the MeerKAT and then the SKA is the biggest ‘big data’ challenge in science – and that challenge starts in South Africa.

“We have a window of opportunity here with the MeerKAT – as a South African instrument – before the world converges on us,” he says. IDIA, a partnership of three South African universities (universities of Cape Town, the Western Cape and Pretoria) is working to build the solutions to the SKA big-data challenge here, so South African researchers can own the data and make the scientific discoveries.

The IDIA resesarch cloud

The IDIA research cloud is the first step toward that goal. IDIA’s computer cloud is tailored to process data on a scale expected from MeerKAT. Beyond just the storage for the raw data, the cloud computing facility allows researchers to use their own tools to process data on unprecedented scales.

Scientists use computer algorithms to build analysis pipelines, turning data into images and to analyse and visualise the data to create new knowledge. On the IDIA research cloud, they can deploy pipelines and analytics that they have already built, and run special programs to visualise and explore the data.

“The size of the data produced by MeerKAT means we need to change the way we do science: the IDIA research cloud will allow us to do that.”

The IDIA cloud technology democratises high-performance computing and makes big data accessible to researchers anywhere. The IDIA cloud software systems are also all built on open source technologies so these tools can be re-used and recreated in other disciplines.

“The size of the data produced by MeerKAT means we need to change the way we do science: the IDIA research cloud will allow us to do that,” says Dr Sarah Blythe of UCT, principle investigator on the LADUMA project of the MeerKAT.

Research on the IDIA Cloud

Without the IDIA research cloud, says postgraduate researcher Wathela Alhassan from UWC, he would be unable to conduct his research. “With the cloud, I can build multiple models with different set ups and architecture and run them at the same time to see which one is better,” he says. “Having access to this much computational power for free is an opportunity I could not even dream of.”

“With the cloud, I can build multiple models with different set ups and architecture and run them at the same time to see which one is better."

Hassan will be using the MeerKAT data to look for unusual or unfamiliar phenomona in the data that may indicate intelligent life on another planet.

“Basically I will be using the IDIA cloud to look for aliens,” he says.

Large science projects like SKA have knock-on effects for the society in which they take place. They stimulate technological advancement, such as the IDIA research cloud, which are then re-used in other disciplines and even industry. A range of scientific disciplines are also facing the big data challenges. Astronomers are among the main users of this cloud right now, but other disciplines – like health and medical research– will benefit from the infrastructure too, hopefully using it to solve some of the wicked problems faced by the world today.