DeepMind AI helps cook up 'novel' compounds – with sides of controversy

Published report 'should be retracted as the main claim of discovery is wrong', UCL chemistry professor tells us

Updated Research by the University of California, Berkeley and Google DeepMind, involving an AI-driven robot producing more than two dozen supposedly novel materials, has been called into question by chemists.

That peer-reviewed DeepMind-Berkeley study, published in November by Nature, described a robotic laboratory operated in America by the Lawrence Berkeley National Laboratory and dubbed the A-Lab.

It was said this facility could automatically synthesize novel compounds devised with the help of DeepMind's GNoMe model plus data from the US government-led Materials Project and various machine-learning processes. The A-Lab required minimal human input, and used artificial intelligence to perform its work, according to its operators.

GNoMe can make two kinds of predictions, according to Google: it can propose potentially novel materials, and has indeed emitted 2.2 million examples of them; and it can predict the stability of these so-called candidate materials. It's believed these structures have the potential to be useful in, say, future electronics.

Over 17 days, according to the published study, the lab's robotic arm made 41 novel materials, 35 of which had been proposed by Berkeley scientists with the help of GNoMe, which made predictions on the stability of the proposed stuff. The automated lab was able to mix and heat various powders to create materials whose structures were then probed using X-ray diffraction.

A separate machine-learning algorithm inspected the laboratory's output and compared the stuff it made to what the machines envisioned it would be to confirm whether the compounds were made successfully as expected. The experiment was billed as an important demonstration that showed how AI-powered robots could help drive scientific discovery.

However, the results are now being disputed. In a fresh non-peer-reviewed paper, seven researchers at Princeton University in the US and University College London in the UK believe that the A-Lab didn't manage to make a single novel inorganic material.

"Unfortunately, we found that the central claim of the A-Lab paper, namely that a large number of previously unknown materials were synthesized, does not hold," the seven wrote in their analysis released via ChemRxiv [PDF]. When they combed through the X-ray diffraction data for each material, they found that most of them have been misclassified.

X-ray diffraction patterns allow scientists to calculate the position of the atoms inside stuff. Different materials will make varying diffraction patterns. Scientists closely inspect the peaks and troughs in the data and compare them to existing patterns to interpret each material's structure.

The data from the A-Lab paper, however, shows that most of the 35 supposedly computer-proposed novel materials actually resembled a mix of already known compounds, while three of them weren't new at all, according to the Princeton-UCL team. These shortcomings stemmed from the method used by the DeepMind-Berkeley group to computationally determine whether a new material had been made or not, Robert Palgrave, a professor of inorganic and materials chemistry at UCL, told The Register.

Researchers at Google DeepMind and UC Berkeley reportedly determined that if each sample made by the robot had a purity level over 50 percent, and if its structure differed from a list of known compounds contained in the ​​Inorganic Crystal Structure Database (ICSD), it should be declared as novel. But that process was unreliable, Prof Palgrave and his colleagues argued.

"On the computational side, they couldn't deal with something called 'compositional disorder,' which is a very important feature of inorganic materials. A crystal is an ordered arrangement of atoms. But even within that order there can be disorder. Imagine you have a set of children's building blocks, all the same size and shape, and they are arranged in a perfectly ordered pattern on the floor. The blocks are like atoms in a crystal," Professor Palgrave told us.

"But now imagine that there are two colors of block, red and blue. We have an ordered pattern of colors, say alternating red, blue, red, blue etc. You might end up with a chess board type arrangement. But it is also possible for the colors to be mixed up randomly. In this case the blocks themselves are ordered, but the colors are disordered."

The chemists believe the initial experiment had not taken compositional disorder into account, and assumed the atoms in each compound made by the A-Lab were ordered when they were actually disordered and already existed in the ICSD. "On the experimental side, they tried to use AI to interpret their experimental data, but it really didn't do a good job. I think AI can certainly do this kind of analysis. I have no idea why they failed, but the outputs are worse than even a novice human would achieve," Prof Palgrave added.

Many of the outputs were poor fits to the diffraction patterns predicted by the AI-based software in the first place, and they cannot be reliably used as proof of a compound's structure or purity, the group said. The results don't necessarily cast doubt on the GNoME project per se. In fact, Prof Palgrave and his colleagues believe that if some of the inorganic crystal structures predicted by GNoME managed to be successfully synthesized, it would result in a novel material.

Yet the compounds made by the A-Lab aren't new, meaning none of GNoME's new materials appear to have been produced yet, they believe. "My own view is that [the paper] should be retracted as the main claim of discovery of new materials is wrong," he told us.

A representative from Google DeepMind declined to comment on the record.

Gerbrand Ceder, a lead author of the original A-Lab paper and Professor of Materials Science and Engineering at UC Berkeley, told The Register in a statement: "The work of Dr Palgrave is not peer reviewed and we believe it has multiple errors in it. We will comment on it in due course, but will do that through the peer reviewed literature." ®

Updated on February 1

Google representatives have been in touch with some clarifications that we're happy to incorporate into this article, and have done so. Primarily: DeepMind stressed that its GNoMe model was not used to propose the materials to be manufactured by the A-Lab, and instead was used to predict their stability.

Secondary: Google has, to us, distanced itself a little from the Berkeley study, telling The Register that the materials produced by the A-Lab were proposed by the university's researchers. The web giant's reps said the Berkeley scientists "checked their predictions using a Google DeepMind tool," ie: GNoMe.

For what it's worth, at the time of the Nature paper going live, Google boasted in an announcement that the Berkeley-DeepMind study "shows how our AI predictions can be leveraged for autonomous material synthesis." Two people at DeepMind, who are listed as co-authors of the Nature paper, are credited for using Google AI in the "filtering pipeline for novel-materials identification."

Again, we're happy to set the record straight in that regard.