Emotion recognition in talking-face videos using persistent entropy and neural networks

Electronic Research Archive http://www.aimspress.com/journal/era ERA, 30(2): 644–660. DOI: 10.3934/era.2022034 Received: 16 November 2021 Revised: 07 February 2022 Accepted: 14 February 2022 Published: 18 February 2022 Research article Emotion recognition in talking-face videos using persistent entropy and neural networks Eduardo Paluzo-Hidalgo∗, Rocio Gonzalez-Diaz and Guillermo Aguirre-Carrazana Department of Applied Mathematics I, University of Seville, Seville, Spain *Correspondence: Email: [email protected]. Abstract: The automatic recognition of a person’s emotional state has become a very active research field that involves scientists specialized in different areas such as artificial intelligence, computer vision, or psychology, among others. Our main objective in this work is to develop a novel approach, using persistent entropy and neural networks as main tools, to recognise and classify emotions from talking-face videos. Specifically, we combine audio-signal and image-sequence information to compute a topology signature (a 9-dimensional vector) for each video. We prove that small changes in the video produce small changes in the signature, ensuring the stability of the method. These topological signatures are used to feed a neural network to distinguish between the following emotions: calm, happy, sad, angry, fearful, disgust, and surprised. The results reached are promising and competitive, beating the performances achieved in other state-of-the-art works found in the literature. Keywords: topological data analysis; persistent homology; persistent entropy; neural networks; audio-visual emotion recognition; talking-face videos 1. Introduction (Facial) Emotion recognition consists of a series of processes to detect human emotions from (facial) human expressions. When people communicate with others, they are constantly sending and receiving nonverbal cues, expressed through body gestures, voice, facial expressions, and physiological changes. Nonverbal cues increase trust, clarity and provide more information supporting what spoken words transmit. A particular emotional state produces certain verbal and nonverbal signals that transmit information regarding personal feelings. Nowadays, (facial) emotion recognition has become an important research area in the fields of computer vision and artificial intelligence due to its potential applications. In general, people express their emotional state (such as joy, sadness, or anger) through facial expressions and vocal tones, and these are features that are often analyzed for emotion recognition. Several factors make emotion recognition in talking-face videos difficult to handle (see, for 645 example [1]). Among others, one of them is that body language cues are not as available to the listener as opposed to having a face-to-face conversation. Besides, rigid movements of the face in a talkingface video can reduce the accuracy of extracting facial features. Another problem is how to measure a ground truth for emotions. Measuring the right emotion at the right time in a talking-face video is, in general, quite subjective and relies entirely on the research setting. Another challenge to take into account is how to deal with facial micro-expressions. In [2], the facial micro-expressions issue is faced using the knowledge distillation approach (see [3]). These spontaneous and low-intensity expressions are the bottleneck of applying deep learning methods due to the huge amount of data needed to train such methods. Regarding computer-aided emotion recognition research works, roughly speaking, they are focused on the use of various input types such as facial expressions [4–6], speech [7–12] and physical signals [13]. Several emotion classification and recognition techniques have been proposed in the past. Some of them used speech prosody contour information to recognize emotions through different classification methods such as artificial neural networks, multichannel hidden Markov models, mixture of hidden Markov models, and Active Appearance Models. Regarding audio emotion recognition, in [14], a preprocessing scheme is proposed to remove the noise from speech signals based on fast Fourier transformation and spectral analysis. The authors evaluated their model using benchmark IEMOCAP and EMODB datasets and obtained a high recognition accuracy, which were 73 and 90%, respectively. The authors in [15] faced the task of audio emotion recognition using a convolutional neural network. Their baseline model included one-dimensional convolutional layers combined with dropout, batch normalization, and activation layer. The proposed framework achieved 71.61% of accuracy for the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS dataset [16]). In [17], a novel approach based on persistent entropy (a computational topology tool) is developed to obtain a single value for the audio signal of a given video of a person expressing emotions. These data were later used as the input of a support vector machine to classify audio signals into eight different emotions, namely, neutral, calm, happy, sad, angry, fearful, disgust, and surprised. The results obtained were close to the existing accuracy of methods with a greater scope such as the ones introduced in [18]. Regarding visual emotion recognition, in [19], a topology-based approach was used to understand and discern existing patterns between emotions in talking-face videos. In that paper, the authors used precomputed landmark points to compute the persistence diagrams (a key tool in computational topology [20]) from the Vietoris-Rips filtration in a given frame. The paper focuses on emotions at instant moments, i.e., no interframe relation is considered in the image sequence. Furthermore, the audio signal is neither used in that approach. The feature obtained is the persistence diagram and pairwise comparisons are made using bottleneck or Wassertein distances. The classification of emotions is not considered in that paper. Regarding multimodal emotion recognition, different approaches have been explored so far. For example, in the H2020 KRISTINA project∗, multimodal computer-aided emotion recognition is used to help in the interaction between health professionals and migrated patients, allowing to overcome linguistic barriers that hinder communication. Besides, the MixedEmotions project†developed an ap- ∗http://kristina-project.eu/en/ †https://cordis.europa.eu/project/rcn/194226 es.html Electronic Research Archive Volume 30, Issue 2, 644–660. 646 plication based on a complete emotional profile of the person’s behavior. It used data from different channels: multilingual text, data sources, audio and video signals, social media, and structured data. The project offered commercial solutions providing an integrated big linked data platform for emotional analysis, using heterogeneous sets of data and addressing the multilingual and multimodality aspects in a robust and large-scale setting. Moreover, hybrid neural networks combining convolutional neural networks and recurrent neural networks have become the state-of-art for multimodal emotion recognition. For example, the authors in [21] proposed an audiovisual-based hybrid network that combines the predictions of five models for emotion recognition in the wild. The overall accuracy of the proposed method achieved 55.61 and 51.15% classification accuracy for the audio-only and video-only dataset, respectively. The authors used the Afew-va Database for Valence and Arousal Estimation In-the-wild introduced in [22]. In general, solutions based on deep learning methods require a large amount of training data and, up to now, the public datasets are limited [3]. In this paper, we consider both the audio signal and the image sequence of a given talking-face video and we use a novel tool introduced in [23], called persistent entropy, to “vectorize” the topological information provided by the persistence diagrams. Such vectors are used to feed a neural network to classify emotions. Specifically, the method incorporates the idea consisting of computing the persistent entropy of the lower-star filtration of the 1-dimensional simplicial complex obtained by a discretization of the audio signal, together with a particular construction of a 3-dimensional cell complex obtained from the image sequence as follows. First, the precomputed landmark points are used to build the 2-dimensional Delaunay triangulation in each frame, and then they are stacked in a particular way to obtain a 3-dimensional cell complex. Later, we compute the persistent entropy of each of the eight filtrations, considering, respectively, the distance to eight fixed planes (two horizontal planes, two vertical planes, and four oblique planes). This way, the method is able to completely capture the movement in the image sequence. The computed topological features together with the features obtained from the audio-signal, make up a 9-dimensional vector, also called the topological signature, of the given talking-face video. Besides, thanks to [24], we are able to prove that topological signatures are stable to small changes in the input audio signal and image sequence. Finally, the topological signatures computed are used to train a neural network to classify emotions. As said before, the use of persistent entropy to compute the topological signature of a talking-face video is supported by its demonstrated stability under small perturbations in the input data [24] and by numerous applications of this technique in other fields such as, for example, pattern recognition [25], complex systems [26], clustering [27], heart-rate-based sleep-wake classification [28], glioblastoma detection [29], and audio emotion recognition [17]. The idea of computing a 3-dimensional cell complex from an image sequence is not new. In [30], this idea was used for gait classification, but there, the 3-dimensional simplicial complex was obtained from the silhouettes of the person walking in the video. That methodology was also used in [31] to monitor human activities at distance and in [32] for gait-based gender classification. The novelty of the method presented here is the computation of a topological signature associated to a talking-face video, combining topological information from the audio signal and the image sequence. This topological signature is obtained by computing the persistent entropy of certain filtrations constructed on specific cell complexes aimed to capture topological changes along the video that characterize the different emotions considered. The topological signatures computed are then used to Electronic Research Archive Volume 30, Issue 2, 644–660. 647 feed a neural network to classify emotions. Let us observe that the neural network considered in this paper is extremely simple because of the low dimension of the input. The contribution of this work is a workflow of emotion recognition, being its stability guaranteed by topology-based theoretical results. The experimentation section shows that our method outperforms several state-of-the-art emotion recognition methods. The paper is structured as follows. The needed background is introduced in Section 2. The description of the proposed method is provided in Section 3. The stability of the method is proven in Section 4. The experimentation made is presented in Section 5, together with comparisons with state-of-the-art methods. Finally, Section 6 provides conclusions and future work ideas. 2. Background In this section, the main concepts of topological data analysis and neural networks, needed to understand our method for facial emotion recognition, are recalled. 2.1. Topological data analysis Topological data analysis has emerged as an important approach to characterize the behavior of datasets using techniques from topology. Tools from topological data analysis, specifically persistent homology, allow assigning shape descriptors to large and noisy data across a range of spatial scales. We will compute such a descriptor to capture the topological changes along the video and these changes will be used to classify emotions. To compute the persistent homology, we first have to provide the input data with a combinatorial structure that reflects the topology of the underlying space where the input data lay. The combinatorial structure used in this work is the one of cell complexes, whose elements in each dimension d, called d-cells, are d-dimensional topological spaces homeomorphic‡to a d-dimensional ball. This way, a 0-dimensional cell is a point (vertex), a 1-dimensional cell is a curve, a 2-dimensional cell is homeomorphic to a disk, and so on. A cell complex K is a collection of cells constructed inductively: 1) The 0-skeleton K(0) (i.e., the set of 0-cells of K) is a set of points in an ambient n-dimensional space Rn. 2) The d-skeleton K(d)is constructed from the (d−1)-skeleton K(d−1) by attaching d-cells via homeomorphisms. From now on, we will assume that the given cell complex Khas a finite number of cells. The boundary set of a d-cell σ∈Kcan be informally defined as the set of (d−1)-cells in the (d−1)- skeleton K(d−1) used to attach the d-cell σ. Successively adding to F={σ}the boundary set of each cell in F, we obtain the set of faces of σ. For example, the boundary set of an edge is its two endpoints (vertices). A d-dimensional cell complex Kis a cell complex satisfying that the dimension of the cell of the higher dimension in Kis d. A subcomplex of a cell complex Kis a subset K0⊂Kwhich itself is still a cell complex. An example of a subcomplex is the closed star of a vertex v, denoted by St v and defined as follows: A cell σis in St vif there exists µ∈Ksuch that σand vare faces of µ. A filtration is an increasing sequence of cell complexes ∅ ⊂ K1⊂K2⊂ · · · ⊂ Kr=K.See an example of a filtration of cell complexes in Figure 1. There are several methods to compute cell complexes and filtration from input data depending on the nature of the data and the purpose of the analysis. In this work, given a talking-face video, we will ‡A homeomorphism is a bicontinuous and bijective function between two topological spaces. Electronic Research Archive Volume 30, Issue 2, 644–660. 648 collect the set of landmark points Sprecomputed in the image sequence of the video. Such landmark points will be embedded in R3, being the last coordinate, the position of the frame where the landmark points are allocated. Let Vsbe the set of points of R3that are closer to s∈Sthan to any other point of S. That is, for s∈S,Vs={x∈R3|d(x,s)≤d(x,s0)∀s0∈S}.The collection of the sets Vsis a covering for R3and it is called the Voronoi decomposition of R3concerning S. The nerve of this covering is a a simplicial complex called the Delaunay triangulation of S. The construction of this complex is costly in high dimensions, although there exist efficient algorithms for computing it when n=2 and n=3. See [33] for more details on Voronoi diagrams and Delaunay triangulation. The filtration considered in this paper is the lower-star filtration (see [20, page 135]). Let us see how to define it. Consider a real-valued function h, also called height function, on a finite set of points V. Suppose Kis a cell complex with set of vertices V. The lower star of v∈Vis defined as the subset of cells of Kfor which vis the vertex with maximum function value, that is, low St v={σ∈ St v:x∈σ⇒h(x)≤h(v)}.Sort the vertices by their function values, in a non-decreasing order, V={v1,v2,...,vr}. The lower-star filtration K1⊂K2⊂ · · · ⊂ Kr=Ksatisfies that Kjis the union of the lower stars of the first jvertices of V, that is, Kj=Si≤jlow St vi, for all j. Figure 1. From left to right, the lower-star filtration K1⊂K2⊂K3=Kobtained using the height function h. The next step is to compute the persistent homology of a filtration K1⊂K2⊂ · · · ⊂ Kr=K[20,34] that tracks the moment twhere a homology class is born and the moment swhere the same class dies leading to a topological descriptor called persistence diagram. Specifically, a d-dimensional homology class of a cell complex Ktis an element of the d-dimensional homology group Hd(Kt) which is defined as follows. First, the d-dimensional chain group Cd(Kt) is obtained by summing up§the d-cells of Kt. Then, the boundary operator is extended to a linear map ∂dfrom Cd(Kt) to Cd−1(Kt) in an obvious way. Since the boundary of the boundary of a cell is always zero, then the image Bd(K) of ∂d+1is a subgroup of the kernel Zd(K) of ∂d. The d-dimensional homology group of Ktis defined as the quotient group Hd(Kt)=Bd(Kt)/Zd(Kt). Then, the given filtration leads a family of homology groups and homomorphisms {Hd(Kt)→Hd(Ks) : t≤s}from which we can track the births and deaths of homology classes. Now, each homology class αthat was born in Hd(Kt) and died in Hd(Ks) can be stored as a point (t,s). The result is a multiset of points in R2called the persistence diagram for the given filtration. The persistence of the homology class αis the difference pers(α)=s−t. In this work, homology classes with infinity persistence correspond to points of the form (t,N+1), where Nis a fixed big positive integer. This way, all points in the persistence diagram have finite coordinates. The §The ground ring considered in this paper is Z/Z2. Electronic Research Archive Volume 30, Issue 2, 644–660. 649 features of higher persistence are represented by the points furthest from the diagonal, while points nearby to the diagonal may be interpreted as topological noise. Finally, we summarize the information described by a persistence diagram in a quantity called persistent entropy (introduced in [23]) which consists in the Shannon entropy of the probability distribution obtained from the given persistence diagram. Specifically, given a filtration and the corresponding persistence diagram Dgm ={(aj,bj) : j∈J}, the persistent entropy of the filtration is defined as E=−Pj∈Jpjlog(pj) where pj=`j L,`j=bj−aj, and L=Pj∈J`j. Let us notice that if pj≤1, then log(pj)≤0, so the persistent entropy is always positive. Intuitively, the persistent entropy measures how different the persistence of the homology classes that appear along the filtration are. 2.2. Neural networks In this paper, we deal with a supervised classification problem where a set of labelled examples are provided with the aim of making predictions for unlabelled points. A widely extended machine learning model for classification problems is neural networks. In general, we could say that a neural network is a mapping Nω,Φ:Rn→Rkthat depends on a set of weights ωand a set of parameters Φ describing the synapses between neurons, layers, activation functions and any other characteristic of its architecture. A good introduction to artificial neural networks was given in [35]. The specific kind of neural network architecture used in this paper is a feedforward neural network composed of a set of neurons hierarchically organized in layers that are fully connected. In this paper, a 9 ×512 ×128 ×64 ×7 fully-connected feedforward neural network will be used (see Figure 2). Neural networks can be seen as directed graphs where the input is transmitted and transformed along the graph using different activation functions such as ReLu, sigmoid, or softmax. In this paper, the ReLU activation functions are used in the hidden layers and the Softmax activation function in the output layer. 512 128 64 Dropout Figure 2. A 9 ×512 ×128 ×64 ×8 fully-connected feedforward neural network composed of an input layer with 9 neurons, three hidden layers with 512, 128 and 64 neurons, respectively, and an output layer with 7 neurons. To train the neural network Nw,Φfor a supervised classification task, we will use a labelled dataset D={(x,cx)}consisting of a finite set of pairs where, for each pair (x,cx), point xlies in Rnand label cx lies in {0,1,...,k}, for some k∈N. During the training process, the set of weights of the neural network is updated trying to minimize a loss function which measures the difference between the output of the network (obtained with the current weights) and the desired output (dictated by the labelled dataset). Electronic Research Archive Volume 30, Issue 2, 644–660. 650 The loss function used in this paper is the cross-entropy loss function which is related to the Kullback- Leibler divergence: Given two probability distributions P(x) and Q(x) over the same random variable x, the cross-entropy is computed as H(P,Q)=−P(x,cx)∈DP(x) log(Q(x)). To iteratively update the weights, the loss-driven training method used in this paper is the Adam algorithm (introduced in [36]) which is a stochastic gradient-based optimization algorithm. The goal of training a neural network is generalization. That is, we want our neural network to learn from the given data and to apply the learnt information to new data. One way to measure the performance of the trained neural network is to split the given dataset into two subsets called the training set and the test set. When the trained neural network reaches high accuracy on the training set but performs badly on new data, we say that there is an overfitting. Among the different approaches to prevent overfitting existing in the literature, in this paper, we will use dropout regularization that consists in randomly invalidating a certain percentage of the neurons of the neural network during the training procedure (consult [37] for more information). 3. Description of the method In this section, we develop an emotion recognition method using persistent entropy and neural networks as the main tools. Overall, the method works as follows. The input data are talking-face videos with precomputed facial landmark points. For each video, we compute a topological feature obtained from the audio signal together with eight topological features obtained from the image sequence, deriving a 9-dimensional vector called the topological signature of the video. The set of topological signatures obtained from the video dataset will then be used to feed a neural network. The summary of the process workflow is outlined in Figure 3. Video audio signal image sequence landmarks raw signal subsampling (9 frames) subsampling (1000 points) 1-dimensional simplicial complex 3-dimensional cell complex 8 filtrations lower-star filtration persistent entropy 8-dimensional vector 9-dimensional vector neural network 1-dimensional vector persistent entropy Figure 3. Summary of the process workflow for emotion recognition used in this paper. Let us explain, step by step, the procedure outlined above. We start by extracting the landmark points on each frame of the input image sequence (see Figure 4). We assume that these landmark points are already precomputed in the given dataset. For each frame, we use the landmark points to compute the 2-dimensional simplicial complex that consists of the Delaunay triangulation of the set of points corresponding to the spatial position of the landmark points. To connect the topological information along the image sequence, the landmark points corresponding to the same part of the face in consecutive frames are joined by an edge. A 2-dimensional cell is obtained when the two endpoints of an edge are joined to the two endpoints of the corresponding edge in the neighbor frame. A 3-dimensional cell is obtained when the vertices Electronic Research Archive Volume 30, Issue 2, 644–660. 651 Figure 4. The landmark points considered in this paper drawn on a face in a video frame extracted from the RAVDESS dataset. of a triangle of the Delauney triangulation associated to one frame are joined with the vertices of the corresponding triangle in the neighbor frame. The output of the steps described above is a 3-dimensional cell complex Kfor each input image sequence, which condenses all gestures the person is making while recording on video. In Figure 5, the 1-skeleton of the 3-dimensional cell complex Kobtained from an image sequence, is pictured. Figure 5. The 1-skeleton of the cell complex Kobtained from an image sequence of a video extracted from the RAVDESS dataset. The next step in this process is to sort the cells of Kin different ways to obtain different filtrations with the aim of capturing the small details that characterize an emotion. In this work, eight different filtrations (two horizontal, two vertical, and four obliques) are used to obtain eight different persistence diagrams (see Figure 6 to have intuitions). The way to define a filtration is as follows: Given a plane π, we define the filter function hπ:K→Rthat assigns to each vertex of Kits distance to the plane π, and to any other cell of K, the maximum distance of its vertices to the plane π. The cells are sorted according to the function values of their vertices, and then, the lower-star filtration Kπassociated with the plane πis computed. Electronic Research Archive Volume 30, Issue 2, 644–660. 652 Facial landmarks extraction from image sequence Cell complex and eight different filtrations Persistence diagrams Figure 6. Illustration of the steps followed to compute the eight persistent entropy values from an image sequence with precomputed landmark points. From left to right: firstly, the landmark points are extracted from the image sequence. Then, a cell complex Kis computed. Eight different filtrations are used to obtain eight persistence diagrams, one for each filtration. Finally, the eight persistent entropy values are computed, one for each persistence diagram. Next, the persistence diagram is computed for each of the eight filtrations. The algorithm used for this step is described in Algorithm 1 with complexity O(n3) in theory but linear in practice [20, page 159]. Algorithm 1: Computing the persistence diagram for a filtration [38]. Input: A filtration ∅=K0⊂K1⊂K2⊂ · · · ⊂ Kn=Kand an ordering of the cells {σ1, . . . , σm} of Ksuch that if i<jthen ind(σi)<ind(σj) where ind(σi)=min{r:σi∈Kr} Output: The persistence diagram Dgm Initialize H=∅, Dgm =∅, and f(σi)=0 for i∈ {1,...,m} for i=1to mdo if f∂(σi)== 0then H∪ {σi}(a new homology class was born) f(σi)=σi Dgm ∪{(ind(σi),∞)} if f∂(σi),0then Let σj∈f∂(σi) such that j== max{ind(µ) : µ∈f∂(σi)} H\ {σj}(an homology class died) foreach x∈K such that σj∈f(x)do f(x)=f(x)+f∂(σi). Dgm \{(ind(σj),∞)}∪{(ind(σj),ind(σi))} The persistent entropy is then computed for each of the eight persistence diagrams. Due to its formulation, persistent entropy can be computed in linear time. As a result, an 8-dimensional vector is obtained for each image sequence. Besides, for each talking-face video, we add a new entry to the 8-dimensional vector computed consisting of the persistent entropy of the lower-star filtration obtained from the 1-dimensional simplicial complex computed from the raw audio signal of the video as it is done in [17]. Electronic Research Archive Volume 30, Issue 2, 644–660. 659 27. X. Wang, F. Sohel, M. Bennamoun, Y. Guo, H. Lei, Scale space clustering evolution for salient region detection on 3d deformable shapes, Pattern Recognit.,71 (2017), 414–427. https://doi.org/10.1016/j.patcog.2017.05.018 28. Y. M. Chung, C. S. Hu, Y. L. Lo, H. T. Wu, A persistent homology approach to heart rate variability analysis with an application to sleep-wake classification, Front. Phys.,12 (2021), 202. https://doi.org/10.3389/fphys.2021.637684 29. M. Rucco, G. Viticchi, L. Falsetti, Towards personalized diagnosis of glioblastoma in fluidattenuated inversion recovery (flair) by topological interpretable machine learning, Electr. Eng. Syst. Sci.,8(2020), 770. https://doi.org/10.3390/math8050770 30. J. Lamar-Leon, R. Alonso-Baryolo, E. Garcia-Reyes, R. Gonzalez-Diaz, Persistent homology-based gait recognition robust to upper body variations, in Processing of the 23rd International Conference on Pattern Recognition, Academic Press, (2016), 1083–1088. https://doi.org/10.1109/ICPR.2016.7899780 31. J. Lamar-Leon, R. Alonso-Baryolo, E. Garcia-Reyes, R. Gonzalez-Diaz, Topological features for monitoring human activities at distance, in Processing of the 2nd International Workshop on Activity Monitoring by Multiple Distributed Sensing,8703 (2014), 40–51. https://doi.org/10.1007/978-3-319-13323-2 32. J. Lamar-Leon, A. Cerri, E. Garcia-Reyes, R. Gonzalez-Diaz, Gait-based gender classification using persistent homology, in Processing of the 18th Iberoamerican Congress on Progress in Pattern Recognition, Image Analysis, Computer Vision, and Apps,8259 (2013) 366–373. https://doi.org/10.1007/978-3-642-41827-3 46 33. C. D. Toth, J. O’Rourke, J. E. Goodman, Handbook of discrete and computational geometry, CRC press, Academic Press, (2017). https://doi.org/10.1201/9781315119601 34. A. Zomorodian, G. Carlsson, Computing persistent homology, Discrete Comput. Geom.,33 (2005), 249–274. https://doi.org/10.1007/s00454-004-1146-y 35. S. S. Haykin, Neural networks and learning machines, Pearson Education, Upper Saddle River, NJ, Academic Press, 2009. 36. D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, (2017). https://arxiv.org/abs/1412.6980 37. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: A simple way to prevent neural networks from overfitting, J. Mach. Learn. Res.,15 (2014), 1929–1958. http://jmlr.org/papers/v15/srivastava14a.html 38. R. Gonzalez-Diaz, P. Real, On the cohomology of 3d digital images, Discret. Appl. Math.,147 (2005), 245–263. https://doi.org/10.1016/j.dam.2004.09.014 39. E. Diener, R. J. Larsen, S. Levine, R. A. Emmons, Intensity and frequency: dimensions underlying positive and negative affect, J. Pers. Soc. Psychol.,48 (1985), 1253. https://doi.org/10.1037//0022-3514.48.5.1253 40. H. Schlosberg, Three dimensions of emotion, Psychol. Rev.,61 (1954), 81. https://doi.org/10.1037/h0054570 Electronic Research Archive Volume 30, Issue 2, 644–660. 660 41. D. Kami´ nska, T. Sapi´ nski, A. Pelikant, Recognition of emotion intensity basing on neutral speech model, in Man-Machine Interactions 3, Springer, 242 (2014), 451–458. https://doi.org/10.1007/978-3-319-02309-0 49 42. S. W. Byun, S. P. Lee, Human emotion recognition based on the weighted integration method using image sequences and acoustic features, Multimed. Tools. Appl.,80 (2020), 35871–35885. https://doi.org/10.1007/s11042-020-09842-1 43. M. F. H. Siddiqui, A. Y. Javaid, A multimodal facial emotion recognition framework through the fusion of speech with visible and infrared images, Multimodal Technol. Int.,4(2020), 46. https://doi.org/10.3390/mti4030046 44. C. Luna-Jimenez, D. Griol, Z. Callejas, R. Kleinlein, J. Montero, F. Fernandez-Martinez, Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning, Sensors,21 (2021), 7665. https://doi.org/10.3390/s21227665 45. E. Ghaleb, J. Niehues, S. Asteriadis, Multimodal attention-mechanism for temporal emotion recognition, in Processng of the IEEE International Conference on Image Processing, Academic Press, (2020), 251–255. https://doi.org/10.1109/ICIP40778.2020.9191019 c 2022 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0) Electronic Research Archive Volume 30, Issue 2, 644–660.