Development of a speech emotion recognizer for large-scale child-centered audio recordings from a hospital environment

Abstract

<p>In order to study how early emotional experiences shape infant development, one approach is to analyze the emotional content of <a href="https://www.sciencedirect.com/topics/social-sciences/logopedics" title="Learn more about speech from ScienceDirect's AI-generated Topic Pages">speech</a> heard by infants, as captured by child-centered daylong recordings, and as analyzed by automatic speech emotion recognition (SER) systems. However, since large-scale daylong audio is initially unannotated and differs from typical speech corpora from controlled environments, there are no existing in-domain SER systems for the task. Based on existing literature, it is also unclear what is the best approach to deploy a SER system for a new domain. Consequently, in this study, we investigated alternative strategies for deploying a SER system for large-scale child-centered <a href="https://www.sciencedirect.com/topics/computer-science/audio-recording" title="Learn more about audio recordings from ScienceDirect's AI-generated Topic Pages">audio recordings</a> from a neonatal hospital environment, comparing cross-corpus <a href="https://www.sciencedirect.com/topics/computer-science/generalization" title="Learn more about generalization from ScienceDirect's AI-generated Topic Pages">generalization</a>, active learning (AL), and <a href="https://www.sciencedirect.com/topics/computer-science/domain-adaptation" title="Learn more about domain adaptation from ScienceDirect's AI-generated Topic Pages">domain adaptation</a> (DA) methods in the process. We first conducted simulations with existing emotion-labeled speech corpora to find the best strategy for SER system deployment. We then tested how the findings generalize to our new initially unannotated dataset. As a result, we found that the studied AL method provided overall the most consistent results, being less dependent on the specifics of the training corpora or speech features compared to the alternative methods. However, in situations without the possibility to annotate data, unsupervised DA proved to be the best approach. We also observed that deployment of a SER system for real-world daylong child-centered audio recordings achieved a SER performance level comparable to those reported in literature, and that the amount of human effort required for the system deployment was overall relatively modest.<br></p>