Material Property Prediction with Transformers

Abstract

The Transformer neural network architecture has had a profound impact on the state of the art in machine learning in numerous disciplines, well beyond its origins in Natural Language Processing. Nevertheless, the application of Transformer models to the material field remains a relatively underexplored avenue. Therefore, we evaluated the Transformer model's capability in utilizing Many-Body Tensor Representation (MBTR) method in prediction of materials’ Highest Occupied Molecular Orbital (HOMO) energy. The dataset selected for this investigation was QM9, a popular dataset that enabled us to conduct comparative analyses of our model’s efficacy against a broad spectrum of prior studies. In this study, we pursued two principal approaches. Initially, we evaluated the performance of the original MBTR representation and the Transformer on the dataset, implementing only minimal modifications to both the model and the representation. Subsequently, we explored a refined MBTR variant, more suitable for the variable sequence length input of the model, which encompasses the distances between atom pairs within a molecule, alongside a reconfigured Transformer designed to integrate encoded chemical symbols of atom pairs as inputs and utilize their distances for positional embeddings. Using the two approaches, we reached the MAE of 0.123 and 0.071, respectively. We find that the Transformer model, designed to process sequential input, is capable of learning to predict from molecular representations of variable length. It outperforms the most effective kernel-based methodologies and is comparable to other recently studied deep neural networks. In conclusion, we illustrate that, with only slight adaptations, Transformers are able to make comparably accurate predictions of materials’ properties.