Progressive loss functions for speech enhancement with deep neural networks

Llombart et al. EURASIP Journal on Audio, Speech, and Music Processing (2021) 2021:1 https://doi.org/10.1186/s13636-020-00191-3 RESEARCH Open Access Progressive loss functions for speech enhancement with deep neural networks Jorge Llombart*, Dayana Ribas, Antonio Miguel, Luis Vicente, Alfonso Ortega and Eduardo Lleida Abstract The progressive paradigm is a promising strategy to optimize network performance for speech enhancement purposes. Recent works have shown different strategies to improve the accuracy of speech enhancement solutions based on this mechanism. This paper studies the progressive speech enhancement using convolutional and residual neural network architectures and explores two criteria for loss function optimization: weighted and uniform progressive. This work carries out the evaluation on simulated and real speech samples with reverberation and added noise using REVERB and VoiceHome datasets. Experimental results show a variety of achievements among the loss function optimization criteria and the network architectures. Results show that the progressive design strengthens the model and increases the robustness to distortions due to reverberation and noise. Keywords: Progressive loss function, Speech enhancement, ResNet, CNN 1 Introduction Most deep neural network speech enhancement (DNNSE) methods act like a monolithic block, where the noisy signal is the input to the architecture and the enhanced signal is the output, while intermediate signals are not easily interpretable. However, SE can also be performed as a gradual improvement process, with a step-by-step speech denoising. In this paradigm, the signal is enhanced progressively at different system stages, by incrementally improving the speech quality at each stage in terms of noise reduction, speech distortion, etc. The incremental SE paradigm has been recently approached through the so-called progressive speech enhancement (PSE) [1–3]. In this mechanism, the network learning process is decomposed in multiple stages, such that the target is progressively optimized. This way, the subproblem solved at each stage can boost the subsequent learning in the next stages. Previous works following this strategy have shown improved results for the progressive architectures compared to usual DNN-SE methods. *Correspondence: [email protected] ViVoLab, Aragón Institute for Engineering Research (I3A), University of Zaragoza, C/ Pedro Cerbuna 12, 50009 Zaragoza, Spain Previous progressive proposals have focused on the incremental signal-to-noise ratio (SNR) reconstruction at different degrees. In [2], a feedforward deep neural network implemented a regression scheme, where the network target was learning an ideal binary mask responsible for improving the SNR three times in 10 dB. The same example was used with different SNR to achieve the progressive enhancement. In [3], the authors extended this work by testing more advanced architectures. Initially, a reproduction of the procedure in [2]usingalongshortterm memory cell (LSTM) showed a degradation of the SE performance with the number of target layers. Then, at each cleaning step, they used additional knowledge from the previous steps, finally achieving an improvement in performance. More recently and motivated by the interpretability of the enhancement process, we have presented a progressive architecture based on wide residual networks [1]. Our main goal was to understand the enhancement process, step by step, by using a visualization probe at each network block. Insights provided by the interpretation of the enhancement process led to the modification of the network architecture, which provided improved results for the SE process. In the proposed architecture, the mean © The Author(s). 2021 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. Llombart et al. EURASIP Journal on Audio, Speech, and Music Processing (2021) 2021:1 Page 2of 16 square error (MSE) of the log-spectral amplitude (LSA) between the enhanced signal and the reference is computed at every network stage and refreshes the backpropagation gradients. Furthermore, the reconstruction error of each block contributes to the optimization loss function with a weighted progressive mechanism. Our preliminary approach to this problem had the intention of just presenting a progressive approach for DNN speech enhancement [1]. Now, this work deeply studies the progressive strategy for DNN-SE. This paper explores the generalization of the training method on two consolidated DNN architectures used for SE tasks: a convolutional neural network (CNN) and a residual neural network (ResNet). This study analyzes two different criteria to implement the progressive paradigm: the weighted progressive (WP) criterion in [1]andanewly proposed uniform progressive criterion (UP). The UP criterion implements the final optimization of the loss function, considering that the reconstruction errors from all blocks contribute in the same way. Moreover, in this work, we consider not only the dereverberation problem but the whole enhancement problem. Also, a wider experimental setup is implemented, including simulated and real datasets. More recent DNN architectures used for SE such as generative adversarial networks (GAN) [4], U-Net [5], or residual hourglass recurrent neural networks (RHR-Net) [6] have demonstrated their capabilities and currently they offer the best results. Despite these architectures could also benefit from the use of the proposed method, in this work, we concentrate on the performance on a selected set of very well-known, simple, and established architectures to show the benefits in terms of performance without negligible increase in computational complexity (very reduced at training time and no computational increase at inference time) of the progressive approach disregarding the specific method or network architecture. The contributions of this work are: •Study of the PSE on two consolidated deep neural network (DNN) architectures: CNN and ResNet. •Assessment of two criteria for progressive loss function optimization: weighted and uniform. •Exploring the space of input features. •Analysis of the progressive mechanism effect on gradients and speech quality measures. The rest of the paper is organized as follows. Section 2 summarizes the antecedents of this work. Section 3goes deeper into the application of the progressive paradigm to the loss function. Section 4describes the experimental conditions. Section 5presents some preliminary results on the vanishing gradient problem, and Section 6analyzes the behavior of the CNN/ResNet architectures when they are using the progressive paradigm by presenting obtained results. Finally, Section 7concludes the paper. 2 Antecedents The architectures considered in this work are CNN and ResNet. In order to adapt these architectures to the progressive paradigm, it is necessary to add additional restrictions and modify the loss function. In the following subsections, we provide an overview of the architecture design and the loss function that will be the base of this work. 2.1 Architecture Architectures based on CNN are capable of exploiting local patterns in the spectrum from both frequency and temporal domains [7,8]. The effect of noise and reverberation appears as a perturbation of the signal spectral shape extended through a specific time-frequency area. The natural structure of the speech signal or the distortion patterns can show correlation in consecutive time-frequency bins in a context. CNN-based architectures effectively deal with this characteristic of the speech signal structure, what makes them appropriate for speech enhancement purposes. CNN has also appeared combined with recurrent blocks to further model the dynamic correlations among consecutive frames [9]. In Fig. 1,weshowatypical structure of a CNN where each architecture block could have different configurations in terms of convolutional layers, batch normalization, or non-linearities. The incorporation of residual connections brought a regularization potential to the CNN approach [10]. ResNet architecture makes use of shortcut connections between neural network layers, allowing systems to handle more depth, with faster convergence and a smaller gradient vanishing effect. Since they can manage deeper networks, they can be more expressive, provide more detailed representations of the underlying structure of the corrupted signal and manage longer contexts. All of this results in more accurately enhanced speech. We show this modification in Fig. 2, where we describe the connection between convolutional blocks in a residual approach. In [1], we added to the ResNet an additional constraint: the architecture kept a constant number of channels along all the blocks of the DNN. The constant number of channels allowed the output reconstruction and a visualization probe at any internal block. The mandatory progressive signal reconstruction forced an incremental process of the SE that tended to improve the robustness of the model. Besides, this architecture uses a weighted composition of reconstruction errors by block to perform the loss function optimization. This way, each block makes partial reconstruction, and the next block has as input a previously enhanced representation of the signal. Llombart et al. EURASIP Journal on Audio, Speech, and Music Processing (2021) 2021:1 Page 3 of 16 Fig. 1 Convolutional neural network (CNN). The convolutional block can have any distribution of convolutional layers and auxiliary layers such as the batch normalization, or non-linearities 2.2 Loss function In [1], we proposed an SE system based on the reconstruction of the LSA of a noisy signal: the audio signal was reconstructed, by means of the overlap-add mechanism, using the enhanced logarithmic output spectrum with the phase of the original noisy speech. The loss function was the classical MSE between the LSA of the reference and the LSA of the enhanced signal, MSE yn,τ,ˆ xn,τ=1 D D−1  d=0yd,n,τ−ˆ xd,n,τ2(1) where Dis the signal input dimension, yd,n,τ,ˆ xd,n,τare the frequency bins of the logarithmic spectrum at the training example nand frame τ.yn,τis the target vector of the clean LSA reference, and ˆ xn,τis the reconstructed vector of the enhanced signal. From our previous experience [1,11,12], instead of using a frame-by-frame loss function, this loss uses the whole input as a sequence. Namely, the base loss function Fig. 2 Residual neural network (ResNet). The convolutional block can have any distribution of convolutional layers and auxiliary layers such as the batch normalization, or non-linearities. The main difference between CNN and ResNet is the residual path in ResNet is the MSE of the LSA over all the examples and sequence length of an update step, JY,ˆ X=1 N N−1  n=0 1 T T−1  τ=0 MSE yn,τ,ˆ xn,τ(2) where Yand ˆ Xare the LSA representation of the training update. Eachexampleisasequenceofalltheframesofthe input signal, where Nisthenumberofexamplesinthe training procedure step, and Tis the number of frames of the example. In order to simplify the training procedure, all the training examples have the same number of frames. Therefore, the training keeps fixing the same segment size, which is obtained by randomly cropping the input signals. This way, any example selected for a training update is an arbitrary segment of the input example. Finally, [1] implements the progressive paradigm modifying the objective loss function composing the MSE Llombart et al. EURASIP Journal on Audio, Speech, and Music Processing (2021) 2021:1 Page 4 of 16 between noisy input LSA and the enhanced LSA at different network levels or blocks. This progressive loss function is a particular case of this paper proposal, and it will be studied in detail in the following section. 3 Speech enhancement This paper aims to study the underlying potential of the PSE paradigm. Previous works have pointed out the performance improvement of the SE task in progressive architecture designs. Beyond these results, this paper brings the hypothesis that the progressive paradigm obtains better SE performance because these mechanisms also refresh gradients during the neural network training. In the following, we will describe the PSE architecture proposed in this paper, which is based on our previous work [1], but additionally includes a set of novelties/contributions designed explicitly for this study. 3.1 Architecture This paper study will be based on two DNN architectures: progressive convolutional neural network (P-CNN) and progressive residual neural network (P-ResNet). Beyond our previous proposal in [1] using the ResNet topology, this paper includes the CNN topology with comparative purposes and to extend the study to generalize the progressive paradigm to different architectures. Figure 3represents the front-end of both architectures. The input signal, x(t), is first windowed, and then, we obtain the logarithm of the absolute value of its short-term Fourier transform (STFT), yielding the LSA X.Wealso obtain the Mel-scaled filter bank (FB), and Mel-frequency cepstral coefficients (MFCC) with different windowing processes to provide additional information to the network, XC. Fig. 3 Front-end. Preprocessing block at beginning of P-CNN and P-ResNet architectures Both architectures keep the same number of channels along all their convolutional blocks. Also, they use the same basic convolutional block (Fig. 4) to remain as comparable as possible. This convolutional block is composed of two successive identical structures. This structure starts with batch normalization, followed by a parametric rectified linear unit (PReLU), and a 1D-Convolutional layer withthesamenumberofchannelsattheinputandthe output. In Fig. 4,Csisthenumberofchannels.The dimension of the kernel (k) is 3 in all convolutions of the architecture. The output of this structure has the same dimensions as the enhanced output. Thus, we can obtain a partially enhanced signal at each block output of P-CNN and P-ResNet. For this work, we used 1D-convolutional layers. Unlike 2D-convolutional layers that combine temporal and frequency dimensions locally, 1D-convolutional layers perform a global combination over all the frequency dimensions in a short-term temporal context. Recent works suggest that when convolutional architectures are employed, Fig. 4 Convolutional block. Basic block for composing P-CNN and P-ResNet architectures Llombart et al. EURASIP Journal on Audio, Speech, and Music Processing (2021) 2021:1 Page 5 of 16 the use of convolutional layers computed through the single temporal dimension are more appropriate for the speech enhancement processing [13,14]. The multiresolution windowing processing of the signal contributes to the dereverberation task, especially when the impulse response is longer than the window length used in the LSA analysis [15]. Xcoutput is only used as input to the first convolutional block as shown in Figs. 1 and 2. The following blocks have the same input and output dimensions to allow the use of the progressive loss function. By providing the MFCC, the network has the possibility of calculating average cepstral representations to help with the channel identification and improve the dereverberation. The filter bank can also play a role in the identification of useful speech structures in a perceptual scale. As we can see in the experiments, their combined use makes a significant improvement. 3.2 Loss function In [1], we designed a neural network to have the same number of channels as the input signal at certain probe points. To induce the desired behavior, we forced the desired enhanced signal to be obtained at these points by adding their reconstruction errors to the training loss, which provided a progressive reduction of the difference between the reference signal and the reconstruction after each block. Unlike the classical layer-wise training, where a stacking technique is used, we train the whole network against the final objective in the proposed method but with the additional constraint that a full reconstruction after each architecture block must be carried out. Our previous work demonstrated that if we do not force the reconstruction after each block, intermediate block outputs are entirely different from our objective and not interpretable. The inclusion of the reconstruction constraint through our loss function allows the visualization of the enhancement procedure. We can choose an intermediate result to reduce the evaluation computational cost depending on the application and help the training procedure to obtain better results. With the proposed loss function, we add the full reconstruction constraint after each convolutional block minimizing the MSE between the clean reference Yand the block output ˆ Xb(Fig. 5). Equation 3shows a general definition of the progressive loss function as a weighted sum over the reconstruction loss of each convolutional block Fig. 5 PSE general architecture for P-CNN and P-ResNet. This figure illustrates the application of the progressive loss that allows to directly represent the output after each block Llombart et al. EURASIP Journal on Audio, Speech, and Music Processing (2021) 2021:1 Page 6 of 16 Table 1 Training datasets description Dataset Timit Librispeech TedLium Files 6299 292329 56704 Speakers 630 2484 698 Speech type Read speech Conference Interface Close microphone Auditorium microphone JPY,ˆ X= B  b=1 Wb·JY,ˆ Xb.(3) Depending on the weights in Equation 3, it is possible to define different progressive loss function criteria. In [1], we proposed the WP loss function and here we also propose the UP criterion. In the next sections, both criteria are experimentally evaluated in combination with P-CNN and P-ResNet. •Weighted progressive (WP): The main weight of the loss function is the final cost, as usual in approximation tasks. Then, the cost of all the architecture blocks is uniformly distributed and added in a weighted sum, JWP Y,ˆ XB=JY,ˆ XB+α1 B B  b=1 JY,ˆ Xb(4) where B is the number of blocks of the architecture. Note that Equation 4is a particular case of the general progressive loss function in Equation 3, where Wb=α/Bfor b=1, ...,B−1and WB=1+α/B. This loss function implements progressive processing along blocks, i.e., every intermediate block reconstructs the enhanced signal. This design forces the enhancement process to be incremental, from slightly to detailed cleaning. In the end, this processing complements the traditional process to obtain the final system output, namely the standard back-propagation of gradients throughout the full architecture (output-input). •Uniform progressive (UP): This loss function proposes a uniform distribution of the block losses along the architecture, JUP Y,ˆ XB=1 B B  b=1 JY,ˆ Xb,(5) which is a special case of Equation 3where Wb=1/Bfor b=1, ...,B. With this strategy, all the outputs have the same impact in the reconstruction. This way, every block can equally contribute to the final loss, and the full architecture makes the same effort in the signal reconstruction. 4 Experimental setup 4.1 Training data For DNN training, we have used three different public datasets: Tedlium [16] from Ted talks; Librispeech [17], audio-books; and Timit [18], a phonetically balanced distributed read speech. These datasets are fully employed, without any partition. See Table 1for the characteristics of the datasets. 4.2 Data augmentation: reverberated and noisy training data Data augmentation using reverberation and additive noise was performed at the training set. For each random training example, there are three transformations (See Table 2 for further details): 1 Impulse responses: We simulated random rooms and source-receiver distances described through the room impulse responses (RIR) using the python package rir-generator1[19]. For the data augmentation loop, there are three different kinds of simulated rooms: small, medium, and large, selected with a probability of 0.5, 0.3, and 0.2. 2 Additive noise: We add some noise, with SNR uniformly sampled between 5 and 25 dB, from the music and noise files in the Musan dataset [20]. Note that among the noise files, there is crowd noise, but there is not any intelligible speech. 3 Time scaling: We randomly select a scale between 0.8 and 1.2. There are signals with no scaling, i.e., the original speed. Some others are slowed down or sped up. 4.3 Evaluation data For evaluation purposes, we use two databases: (1) REVERB [21] and (2) VoiceHome v0.2 [22]andv1.0 [23]. REVERB is divided in a development set (REVERBDev), generally used for evaluating intermediate results during the study, and an evaluation set (REVERB-Eval), for confirming the results and evaluation of the system. VoiceHome evaluates the system in a realistic domestic environment with noise and reverberation. So, with these twodatabases,wecanseparatetwoconditions: 1https://github.com/Marvin182/rir-generator Llombart et al. EURASIP Journal on Audio, Speech, and Music Processing (2021) 2021:1 Page 7 of 16 Table 2 RIR and noise for training data augmentation Room impulse responses Small Medium Large Probability 0.5 0.3 0.2 Size (x,y,z)[m] x∼U(1, 6),y∼U(1, 6),z∼U(2, 3.5)x∼U(6, 10),y∼U(6, 10),z∼U(3, 5)x∼U(10, 20),y∼U(10, 20),z∼U(4, 6) RT60[s] RT60 ∼U(0.1, 0.25) Distance[m] 0.5, 1.0, 1.5, 2.0, 2.5 Microphone type Bidirectional, hypercardiodid, cardioid, subcardoid, omnidirectional Noise Music 659 files Noise 929 files SNR [dB] SNR ∼U(5, 25) Simulated data Part of the REVERB dataset corresponds to simulated conditions. They are speech samples from the WSJCAM0 corpus [24]combinedwiththreekindsof RIR: small, medium, and big room (RT60 =0.25, 0.5, 0.7s). For each one, there are two source-mic distances: far (2m) and near (0.5m). Also, a stationary noise was added from the same rooms (SNR =20dB). For this study, we only use the first channel of the eight available. We also add five noises (SNR =0, 5, 10, 15, 20, and 25dB)to all signals at the simulated condition of REVERB. These are babble noise, cafe environment noise, music, street environment with lot of traffic, and noise captured inside a moving tram. Real data We used two evaluation sets with real conditions: the real part in REVERB and VoiceHome dataset (v0.2 and v1.0). REVERB was recorded in a meeting room with RT60 =0.7sat two distances: far (2.5 m)and near (1m),fromMC-WSJ-AV[25]. VoiceHome corresponds to a realistic domestic environment with everyday noises like a vacuum cleaner, dish-washing, or sound of TV shows. 4.4 Speech quality measures To measure the level of denoising and dereverberation achieved by the PSE method, we estimate the segmental SNR [26] and the speech-to-reverberation modulation energy ratio (SRMR) [27,28]. In these metrics, the higher the values, the better speech quality. However, it is well-known that the SE processing might generate distortion on the output speech. Therefore, for the simulated dataset, we also measure the distortion between the clean reference and enhanced speech using the loglikelihood ratio (LLR) [29]. In this case, lower values mean less distortion, so the better quality of the speech. The combination of both speech quality viewpoints, i.e., the trade-off between noise/reverberation reduction and distortion, provides a general assessment of the SE method performance. This way, the best enhancement system is the one which improves SNR or SRMR, but retains the distortion, in this case, measured with LLR, as low as possible. Additionally we use the well-known PESQ measure [30] for simulated data. PESQ measure is in range 0-5 where the higher the better performance. 4.5 Neural network configuration The input provided to the CNN, ResNet, P-CNN, and P-ResNet architectures consists of the logarithm of the magnitude of the 512-STFT of the corrupted signal, sampled at 16 kHz. The STFT is computed every 10 ms for a 25 ms sliding Hamming window. We also concatenate the Mel-Scaled Filter-bank and the MFCC as auxiliary inputs, with filter bank sizes 32, 50, and 100, every 10 ms. MFCC are computed using the discrete cosine transform (DCT) without truncation. However, each frequency resolution has a different sliding Hamming window of 25 ms, 50 ms, and 75 ms respectively. These auxiliary features provide different frequency and temporal resolutions, which can benefit the speech enhancement process [15]. Taking into account that the LSA dimension is 512, the overall input size is 876. For all the experiments, we use adaptive moment estimator (Adam) as the update function. Each layer has 512 neurons to follow the philosophy of maintaining unaltered the number of channels along the architecture. The training consists of 900 epochs. For each epoch, 10,000 input files are randomly selected from the training set. As long as there are unused training examples, no file can be selected more than once. Batch normalization moving parameters are blocked after epoch 700. For the JWP loss function, we use α=0.1 as in [1], which provided the best SRMR value on REVERB-Dev. 5 Preliminary gradient study This section presents a preliminary study of the behavior of the gradient to explore how the injection of new fresh gradients at different architecture levels improve Llombart et al. EURASIP Journal on Audio, Speech, and Music Processing (2021) 2021:1 Page 8 of 16 the training procedure. When gradients back-propagate through a large number of layers, they tend to lose energy. Thus, their ability to move weights of the layers near to the input is reduced. The proposed PSE method feeds a fresh and stronger gradient after each block to move the weights of each layer. In order to check this, we design an experiment to observe the energy of the gradients that modify the weights of the first convolutional block during the 100 first optimization updates. This procedure is repeated 100 times with different weight initializations to observe the variance among different starts and the variation of this gradient energy during optimization. Figure 6presents the results obtained for P-CNN and PResNet architectures, for non-progressive baselines, and for each proposed progressive criteria. There is a noticeable difference in the behavior of the two structures. In PCNN, there is a significant difference among the gradient energy of each compared system. The lower energy corresponds to the baseline architecture, the one without any progressive assumption. On the other hand, the progressive mechanisms show a significant lifting of the gradient energy. These boosted gradients have more strength to move the weights allowing a better learning at inner layers of the whole architecture. In contrast, in P-ResNet, there is no relevant difference between the gradient energy of the progressive techniques and that of the no progressive baseline at the first convolutional block. Consider that P-ResNet is an architecture designed to deal with the vanishing problem, and thanks to residual connections, the gradients have a shortcut to propagate up to the first layers without vanishing. In this case, injecting new gradients does not push much more the previous gradients. However, the new gradients are more accurate because they directly come from the target evaluation at the output of each architecture block. 6 Results and discussion 6.1 Analysis of alternatives for the DNN input In this section, we present a study to asses that the combined use of complementary inputs to the corrupted LSA may improve the performance of the system. We use multiresolution in the MFCC and FB inputs as described in Section 4.5, but we perform an ablation study about the use of each feature type. For this study, we focus on the dereverberation performance of the P-ResNet with Fig. 6 Mean and standard deviation (shaded area) of the log-energy of the gradients of 100 random network initialization during the 100 first mini-batches in training. The log-energy is measured on the nearest block to the input Llombart et al. EURASIP Journal on Audio, Speech, and Music Processing (2021) 2021:1 Page 9 of 16 Table 3 Evaluation of the use of complementary information at the input of the P-ResNet with WP architecture over the REVERB-Dev dataset in terms of reverberation measured with SRMR Complementary info. Real condition Simulated condition Without 6.55 7.99 FB 7.25 8.31 MFCC 7.07 8.44 FB + MFCC 7.14 8.41 Bold text remarks on the best result per condition and italic text the second best WP over the REVERB-Dev dataset in real and simulated conditions. Table 3shows that the best results in simulated conditions are attained using only MFCC, but for real conditions they are obtained with FB features. On average, the combined use of both features, FB and MFCC, provides the best performance, especially compared to the use of LSA without any auxiliary inputs. 6.2 Architecture depth analysis SE progressive methods use a sequence of steps to perform the enhancement. We have to determine the number of steps or the number of blocks that composes the architecture. Table 4shows the architecture depth study in terms of SRMR over the REVERB-Dev dataset. This study shows the results for simulated and real conditions and the average of both. Result indicate that the configuration with 16 blocks achieves the best performance for all the evaluated conditions. Note how progressive systems can achieve high SRMR, both for simulated and real conditions. This consistency among different conditions demonstrates how the progressive strategy can provide a better generalization to the DNN training. For CNN topology, the reference system in real conditions quickly degrades the performance with the depth of the architecture. Besides, results for P-CNN with UP are better than the CNN reference system, i.e., P-CNN with UP does not degrade as fast as CNN reference system as depth increases. For ResNet topology, the availability of residual connections works well with a high number of blocks. For instance, the results of the ResNet reference system achieve the best performance on simulated conditions with the deeper architecture (32 blocks). However, note that in real conditions, the ResNet reference system achieves the best result with 8 blocks versus the 32 blocks for simulated conditions. Nevertheless, P-ResNet Table 4 Speech quality in terms of SRMR for simulated and real reverberated speech samples through architecture depth for REVERB-Dev dataset. The last rows represents the mean and standard deviation along the experiments presented for each column Reference systems Progressive systems Condition Blocks depth CNN ResNet P-CNN with WP P-CNN with UP P-ResNet with WP P-ResNet with UP Simulated 8 7.33 8.23 6.49 7.53 8.31 7.91 16 7.60 8.27 8.96 7.70 8.41 8.05 24 8.87 8.14 6.18 8.09 8.03 8.02 32 7.01 8.56 7.65 7.41 7.98 7.78 Real 8 6.05 6.82 4.90 6.32 7.06 6.91 16 5.98 5.81 3.74 7.26 7.14 6.85 24 4.76 5.77 2.07 6.90 6.53 6.91 32 3.35 6.33 2.33 6.34 5.97 6.62 AVG5±STD 86.69±0.64 7.52±0.70 5.69±0.79 6.92±0.60 7.68±0.62 7.41±0.50 16 6.79±0.81 7.04±1.23 6.35±2.61 7.48±0.22 7.77±0.63 7.45±0.60 24 6.81±2.05 6.97±1.16 4.12±2.05 7.49±0.59 7.28±0.75 7.46±0.55 32 5.18±1.83 7.44±1.11 4.99±2.66 6.87±0.53 6.97±1.00 7.20±0.58 Bold values show the best result for each condition Llombart et al. EURASIP Journal on Audio, Speech, and Music Processing (2021) 2021:1 Page 16of16 21. K. Kinoshita, M. Delcroix, T. Yoshioka, T. Nakatani, E. Habets, R. Haeb-Umbach, V. Leutnant, A. Sehr, W. Kellermann, R. Maas, S. Gannot, B. Raj, in Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA-13). The REVERB challenge: a common evaluation framework for dereverberation and recognition of reverberant speech (IEEE, New Paltz, 2013), pp. 1–4 22. N. Bertin, E. Camberlein, E. Vincent, R. Lebarbenchon, S. Peillon, É. Lamandé, S. Sivasankaran, F. Bimbot, I. Illina, A. Tom, et al,inInterspeech 2016. A French corpus for distant-microphone speech processing in real homes, (San Francisco, 2016), pp. 2781–2785 23. N. Bertin, E. Camberlein, R. Lebarbenchon, E. Vincent, S. Sivasankaran, I. Illina, F. Bimbot. VoiceHome-2, an extended corpus for multichannel speech processing in real homes, vol. 106, (2019), pp. 68–78 24. T. Robinson, J. Fransen, D. Pye, J. Foote, S. Renals, in IEEE International Conference on Acoustic, Speech and Signal Processing (ICASSP).WSJCAM0:a British English speech corpus for large vocabulary continuous speech recognition (IEEE, Detroit, 1995), pp. 81–84 25. M. Lincoln, I. McCowan, J. Vepa, H. K. Maganti, in Proceedings of the 2005 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU-05). The multi-channel Wall Street Journal audio visual corpus (MC-WSJ-AV): specification and initial experiments (IEEE, Philadelphia, 2005), pp. 357–362 26. C. Kim, R. M. Stern, in Ninth Annual Conference of the International Speech Communication Association (Interspeech 2008). Robust signal-to-noise ratio estimation based on waveform amplitude distribution analysis, (Brisbane, 2008) 27. T. H. Falk, C. Zheng, W. Y. Chan, A non-intrusive quality and intelligibility measure of reverberant and dereverberated speech. IEEE Trans. Audio Speech Lang. Process. 18(7), 1766–1774 (2010) 28. J. F. Santos, M. Senoussaoui, T. H. Falk, in Proc. Int. Workshop Acoust. Signal Enhancement (IWAENC 2014). An updated objective intelligibility estimation metric for normal hearing listeners under noise and reverberation, (Antibes - Jaun les Pins, 2014), pp. 55–59 29. P. C. Loizou, Speech quality asssessment. in: multimedia analysis, processing and communications. (Springer, Berlin, 2011), pp. 623–654 30. A. W. Rix, J. G. Beerends, M. P. Hollier, A. P. Hekstra, in 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221), vol. 2. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs (IEEE, Salt Lake City, 2001), pp. 749–752 31. L. Drude, J. Heymann, C. Boeddeker, R. Haeb-Umbach, NARA-WPE: a Python package for weighted prediction error dereverberation in Numpy and Tensorflow for online and offline processing, (Stuttgart, 2018), pp. 1–5 32. T. Nakatani, T. Yoshioka, K. Kinoshita, M. Miyoshi, B.-H. Juang. Speech dereverberation based on variance-normalized delayed linear prediction, vol. 18, (2010), pp. 1717–1731 Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.