The Generalized Complex Kernel Least-Mean-Square Algorithm

Depósito de investigación de la Universidad de Sevilla https://idus.us.es/ This is an Accepted Manuscript of an article published by IEEE Transactions on Signal Processing on 15/10/2019, available at: https://doi.org/10.1109/TSP.2019.2937289 “© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other Works” 1 C { } The Generalized Complex Kernel Least-Mean-Square Algorithm Rafael Boloix-Tortosa*, Member, IEEE, Juan Jose´ Murillo-Fuentes, Senior Member, IEEE, Sotirios A. Tsaftaris, Senior Member, IEEE Abstract—We propose a novel adaptive kernel-based regression method for complex-valued signals: the generalized complexvalued kernel least-mean-square (gCKLMS). We borrow from the new results on widely linear reproducing kernel Hilbert space (WL-RKHS) for nonlinear regression and complex-valued signals, recently proposed by the authors. This paper shows that in the adaptive version of the kernel regression for complexvalued signals we need to include another kernel term, the socalled pseudo-kernel. This new solution is endowed with better representation capabilities in complex-valued fields since it can efficiently decouple the learning of the real and the imaginary part. Also, we review previous realizations of the complex KLMS algorithm and its augmented version to prove that they can be rewritten as particular cases of the gCKLMS. Furthermore, important conclusions on the design of the kernels are drawn that help to greatly improve the convergence of the algorithms. In the experiments, we revisit the nonlinear channel equalization problem to highlight the better convergence of the gCKLMS compared to previous solutions. Also, the flexibility of the proposed generalized approach is tested in a second experiment with non-independent real and imaginary parts. The results illustrate the significant performance improvements of the gCKLMS approach when the complex-valued signals have different properties for the real and imaginary parts. Index Terms—LMS, complex-valued, RKHS, kernel methods. I. INTRODUCTION OMPLEX-VALUED signals model many systems in diverse applications such as electromagnetism, telecommunications, optics or acoustics, among others. Complex-valued signal processing is thus of fundamental interest as it provides a natural way to represent some signals and transformations involved in those systems. While the linear case has been widely studied (see for example [1] and references therein), nonlinear processing still remains an open problem. Nonlinear processing of complex-valued signals has been tackled, among others, from the point of view of neural networks [2], [3], nonlinear adaptive filtering [4], or reproducing kernel Hilbert spaces (RKHS) [5]. This latter field is gaining increasing interest within the signal processing community as it provides a simple but elegant way to treat nonlinearities. Complex R. Boloix-Tortosa, and J.J. Murillo-Fuentes are with the Dep. de Teor´ıa de la Sen˜al y Comunicaciones, Escuela Te´cnica Superior de Ingenier´ıa, Universidad de Sevilla, Camino de los Descubrimientos sn, 41092 Sevilla, Spain. e-mail: rboloix,murillo @us.es. S. A. Tsaftaris is with the School of Engineering, The University of Edinburgh, Edinburgh EH8 3FB, U.K., and also with The Alan Turing Institute, London NW1 2DB, U.K. Thanks to the Spanish Government (Ministerio de Econom´ıa y Competitividad, TEC2016-78434-C3-02-R, and Ministerio de Educacio´n, Cultura y Deporte, Subprograma Estatal de Movilidad (PRX18/00523), del Plan Estatal de I+D+I) and European Union (FEDER) for funding. kernel-based algorithms have been lately proposed for regression [6], [7], [8], kernel principal component analysis [9] or classification [10]. Regarding complex-valued regression within the RKHS framework, we have recently highlighted in [11] the need of a new term: the pseudo-kernel. We redefined the kernelbased regularized least squares regression to include the pseudo-kernel, and the resulting structure resembles that of the widely linear (WL) solutions, being capable of learning any complex-valued function effectively. As discussed in [11], the need for a pseudo-kernel can be justified in cases where the real and imaginary parts are correlated and learning them independently is, at best, suboptimal. Also, a pseudo-kernel is needed when the real and imaginary parts are not best represented by the same kernel, i.e., the same measure of similarity. Furthermore, we analyzed in [11] the structure of the kernel and pseudo-kernel, and discussed how to design these functions, and when should they be real or complexvalued. As a result, two important remarks were made. First, if the real and imaginary parts of the output are independent, then the kernel and pseudo-kernel should be real-valued. Second, if the real and imaginary parts of the output have different properties in terms of similarity, the pseudo-kernel is needed. On the contrary, the pseudo-kernel vanishes if the real and imaginary parts of the output are independent but have the same properties in terms of similarity, i.e., the same kernel can be used for the real and imaginary parts. The complex kernel least-mean-square (CKLMS) allows for an adaptive version of the RKHS techniques. In the literature, there are several proposals for CKLMS algorithms. The authors in [6] address the problem of adaptive filtering of complex signals and calculate the gradient of cost functions by using Wirtinger’s derivatives. Two alternatives are described. The first alternative, denoted by CKLMS1, uses real kernels developed by means of the complexification technique of real RKHSs. The second one, the CKLMS2, proposes the use of complex kernels, in particular, the complex Gaussian kernel [10]. With these two alternatives, they develop two realizations of the kernel least-mean-square (KLMS) algorithm [12] for complex signals. The same complex Gaussian kernel is also adopted in [13] and in [14]. Augmented or WL filters consider both the original values of the signal data and their conjugates [15]. In [14], the authors introduce WL adaptive filters to develop an augmented version of the complex KLMS algorithm, the ACKLMS. In the light of the theoretical framework in [11], in this paper we review these results on the CKLMS to propose a new approach, as follows. 2 · ∈ Σ ∈{ ··· } Y Y × Y ∈ Y X Y Y → Y ∈ X ⟨ ⟩ X × X → L Y L Y ⟨ ⟩ ⟨ ⟩ ⟨ ⟩ R { } Y H ∈ X ∈ Y H X Previous augmented or WL versions of the complex KLMS use limited feature mappings. The KLMS algorithm [12] considers a linear input-output mapping but on the transformed inputs by using the feature map, y ˆ = Φ( x ) , w . For the ACKLMS algorithm [14] they propose to use a WL model with two linear terms, one with the feature map transformation and the other with the conjugate of this transformation: y ˆ = Φ( x ) , w + Φ( x ) ∗ , v . We propose a richer way to represent this model leading to improved results on the adaptive regression task for complex-valued signals. This is the representation of a complex-valued function as a twodimensional real-valued vector function obtained by stacking the real part on top of the imaginary part. From the theory of kernels for learning vector-valued functions [16] we are able to propose a suitable feature map transformation in this composite representation. By exploiting this composite feature map, we can set a linear input-output mapping on the transformed inputs to derive the KLMS algorithm for the composite representation. This is denoted using bold upper-case letters. For matrix A , [ A ] l,q is its (l, q) entry. To denote the i-th sample of a vector or signal we use, respectively, a (i) and a(i) . a is the real part of a . Transpose operation is represented by T, while H represents the Hermitian and ∗ complex conjugation. E [ ] is the expectation operator. II. RKHS OF COMPOSITE VECTOR-VALUED FUNCTIONS A complex function f ( x ) = fr( x ) + j fj( x ) can be represented as a composite vector-valued function f R ( x ) = [fr( x ) fj( x )] T R 2 , also known as the dual real channel (DRC) formulation, by stacking its real part on its imaginary part. The definition of the RKHS for vector-valued functions [16] parallels the one for scalar functions [20], with the main difference that the reproducing kernel is now matrix-valued [21], [16]. Let be a Hilbert space of functions f on a set with values in . is a RKHS when for any x and any y the linear functional which maps f to (y, f (x)) is continuous the second contribution of the paper. In this new composite KLMS algorithm we prove that the kernel should be matrixon H [16]. Here, (·, ·) Y Y represents the inner product in the valued. Once the algorithm has been developed in composite representation we develop its complex value representation, which is the third and central contribution of this paper. This is an augmented or WL version of the CKLMS. To avoid confusion with previous proposals we denote it by generalized CKLMS (gCKLSM). It has both a kernel and a pseudo-kernel, in accordance with the general theory for complex RKHS in [11]. This representation exhibits better reproducing capabilities than previous approaches [17], [18], [19], [14]. Furthermore, we show that these previous proposals are particular cases of the gCKLMS. The right design of the kernel and the pseudo-kernel is important to properly model the system. As a final contribution we analyze different options and explain their implications. The kernel and pseudo-kernel in the gCKLSM have the same structures introduced in [11], this is our starting point to enhance the performance of the gCKLMS. These results can also be used to improve previous proposals. This paper is organized as follows. We devote Section II to review the theory of learning in RKHS of vector-valued functions [16] as a way to find a suitable feature map transformation. In Section III we use the feature map to develop the KLMS algorithm for the composite representation. The formulation for the gCKLMS algorithm is found in Section IV. In this section we also show the equations for the kernel and pseudo-kernel terms. In Section V we compare the gCKLMS with other complex KLMS algorithms in the literature to show that they are particular cases of the gCKLMS. Experiments are included in Section VI, where the gCKLMS algorithm is tested first in the context of a nonlinear channel equalization task, and then in the learning of samples of a filtered random process. These experiments show that the gCKLMS outperforms other KLMS algorithms because it takes advantage of having both the kernel and the pseudo-kernel. We end the paper with some conclusions in Section VII. In the notation used throughout the paper, bold lowercase letters are used to denote vectors, while matrices are Hilbert space Y, while ⟨·, ·⟩H is the inner product in H. From the Riesz Lemma, for every x ∈ X and y ∈ Y there is a linear operator K x : Y → H , such that ( y , f ( x )) Y = K x y , f H. Let us now introduce the linear operator K ( x , x ′ ) : , for every x , x ′ , defined by K ( x , x ′ ) y := ( K x ' y )( x ) . We say that K : ( ) , where ( ) denotes the set of all bounded linear operators from to itself, is a matrix-valued kernel [16] (or operator-valued kernel if is not finite dimensional [22]) if it satisfies the following properties for every x , x ′ ∈ X : (a) For every y , y ′ ∈ Y , we have ( y , K ( x , x ′ ) y ′ ) Y = ⟨ K x y , K x ' y ′ ⟩ H. (b) K ( x , x ′ ) = K ¯ ( x ′ , x ) , and K ( x , x ) ∈ L + ( Y ) , where K ¯ denotes the adjoint and L+(Y) the set of all positive semidefinite bounded linear operators, i.e., ( y , K ( x , x ) y ) Y ≥ 0 for any y . (c) For any positive integer m, we have that l,q 1, ,m ( y q , K ( x q , x l ) y l ) Y ≥ 0 , for any x l , x q ∈ X , y l , y q ∈ Y . Proof of these properties can be found in [16]. Also, it can be shown that if K is a kernel then there exists a unique (up to an isometry) RKHS of functions from to which admits K as the reproducing kernel. In the case of = R2, the kernel function K takes values as 2 2 matrices and, from property (a), the matrix elements can be found as: [ K ( x , x ′ )] l,q = ⟨ K x e l , K x ' e q ⟩ H , (1) where el, eq are the standard coordinate bases in R2, for l, q ∈ { 1, 2 } . A. Feature map We next define a suitable feature map representation for the matrix-valued kernel that will be later useful in deriving the gCKLMS algorithm. 3 ∈ X × W Y ∈ X Σ — − Σ Y ∈ W { } ∈ ∈ | − | | | Σ Σ = Φ ( x ( l )) Φ ( x ( l )) = r j ∈ ∈ Y × W I − j I n Σ 1 Σ = φ ( x ) w = r Φ T j ( x ) n ∈ Every kernel K admits a feature map representation. A feature map is a continuous function φ : X → L(Y, W), where L(Y, W) denotes all bounded linear operators from Y into the feature Hilbert space W [22]. If φ ¯ ( x ) is the adjoint of φ ( x ) , it is in L ( W , Y ) , then K ( x , x ′ ) = φ ¯ ( x ) φ ( x ′ ) , (2) and the update equation for w using the stochastic gradient yields w (i) = w (i − 1) + 2µ φ ( x (i)) e R (i). (8) If we set w (0) = 0 , the repeated application of the weightupdate equation (8) yields for any x , x ′ . In the case of finite dimensional Hilbert spaces = R2 and = R m , relative to standard basis of both spaces φ ( x ) is a m 2 matrix. Each entry of this matrix, [ φ ( x )] p,q = φ pq ( x ) is a scalar-valued continuous function of x , and each entry of the kernel is i w (i) = 2µ φ ( x (l)) e R (l). (9) l=1 At instant i the output can be estimated using the last updated weights, w ( i 1) , as y ˆ R ( i ) = f R ( x ( i )) = φ T ( x ( i )) w ( i 1) . Therefore, the input-output operation of the composite KLMS algorithm can be expressed as [ K ( x , x ′ )] l,q = φ il ( x )φ iq ( x ′ ). (3) i ∈{ 1, ··· ,m } y ˆ R ( i ) = φ T ( x ( i )) w ( i − 1) i − 1 Note that when = R, then φ ( x ) , but this is not the case here. III. THE COMPOSITE KLMS ALGORITHM Consider the training sequence of input-output pairs ( x (1), y(1)), ..., ( x (N ), y(N )) where y(n) C and x (n) Cd. The goal is to uncover the underlying complex-valued function f (x(i)) based on these examples, so that to minimize the mean square error J = E [ y(i) f ( x (i)) 2 ] = E [ e(i) 2 ] . By using the composite notation, this can be written as J = E [ | e(i) | 2 ] = E [(y r (i) − f r ( x (i))) 2 + (y j (i) − f j ( x (i))) 2 ] T = 2µ φ T ( x (i)) φ ( x (l)) e R (l) l=1 i − 1 = 2µ K ( x (i), x (l)) e R (l), (10) l=1 where the matrix-valued kernel yields: K ( x (i), x (l)) = φ T ( x (i)) φ ( x (l)) Φ T r ( x ( i )) Φ T j ( x ( i ) Φ T r ( x ( i )) Φr ( x ( l )) Φ r T ( x ( i )) Φj ( x ( l )) Φ T j ( x ( i )) Φr ( x ( l )) Φ T j ( x ( i )) Φj ( x ( l )) = E [( y R (i) − f R ( x (i))) ( y R (i) − f R ( x (i)))], (4) k rr ( x (i), x (l)) k rj ( x (i), x (l)) The least-mean-square (LMS) algorithm would consider a linear input-output mapping, i.e., f ( x (i)) = w H x (i) , and compute the weight vector w adaptively using stochastic gradient descent updates [23]. However, instead of a direct linear input-output mapping, the KLMS [12] is performed on the transformed inputs by using the feature map. We propose here to use the composite notations and the theory for RKHS of composite vector-valued functions described in the previous section. Therefore, we use the feature map φ : X → L(Y, W) and set f R ( x ) = φ ¯ ( x ) w , where w ∈ W . Note that in the general case W could be an infinite Notice that this kernel matrix follows the structure introduced in [11] for the WL-RKHS, and is composed of four scalar real functions. IV. THE PROPOSED GENERALIZED COMPLEX KLMS ALGORITHM Any real-valued composite vector representation y R = [ y r T y j T ] T R 2n of any complex-valued vector y = y r + j y j C n , can be related to the complex augmented vector y = [ y T y H ] T C 2n representation, which is obtained by stacking y on top of its complex conjugate y ∗. The relation dimensional Hilbert space. For the particular case of = Rm, since = R 2 then φ ( x ) = [ Φ r( x ) Φ j( x )] is an m 2 matrix, where Φr(x) and Φj(x) are its first and second column, respectively, and φ ¯ ( x ) = φ T ( x ) : is y = T n y R , where T n = I j I ∈ C 2n × 2n , (12) f r ( x ) T Φ T ( x ) which is a unitary matrix up to a factor of 2: T n T H = The objective is now the minimization of We can now apply this relation to (10) to calculate: J( w ) = E y R (i) − φ T ( x (i)) w T y R (i) − φ T ( x (i)) w . y ˆ( i ) = y ˆ( i ) y ˆ ∗ ( i ) = T 1 y ˆ R ( i ) = T 1 2 µ i − 1 l=1 K ( x (i), x (l)) e R (l) It is easy to show that the gradient is (6) = 2µ i − 1 T 1 K ( x (i), x (l)) T H T 1 2 e R (l) ∂J( w ) ∂ w = − 2 E h φ ( x (i)) y R (i) − φ T ( x (i)) w i l=1 i − 1 = µ K A ( x (i), x (l)) e (l). (13) = − 2 E [ φ ( x (i)) e R (i)], (7) l=1 T H T n = 2 I , where I is the identity matrix. fj( x ) k jr ( x (i), x (l)) k jj ( x (i), x (l)) where f R ( x ) = [fr( x ) fj( x )] T and y R = [yr yj] T. Σ = . (11) f R ( x ) = w . (5) 1 4 Σ R { } Σ Σ Σ l=1 i − 1 Here we have the augmented error vector e (l) = T 1 e R (l) = [e(l) e ∗ (l)] T, and the augmented kernel matrix H At this point it is interesting to note that (20) and (19) are the same. If we take e(l) as a common factor in (20), it follows: i − 1 K A ( x (i), x (l)) = T 1 K ( x (i), x (l)) T 1 Σ ∗ k ( x ( i ) , x ( l )) k ˜ ( x ( i ) , x ( l )) y ˆ( i ) = µ e ( l ) ( k ( x ( i ) , x ( l )) + k ( x ( i ) , x ( l ))) where by using (11) the complex kernel and complex pseudokernel can be identified, respectively, as k( x (i), x (l)) = k rr ( x (i), x (l)) + k jj ( x (i), x (l)) + j (k jr ( x (i), x (l)) − k rj ( x (i), x (l))) , (15) k ˜ ( x ( i ) , x ( l )) = k rr ( x ( i ) , x ( l )) − k jj ( x ( i ) , x ( l )) + j (kjr( x (i), x (l)) + krj( x (i), x (l))) . (16) Notice that this kernel and pseudo-kernel follow the structure introduced in [11]. The first entry of y ˆ( i ) in (13) yields the proposed generalized complex KLMS (gCKLMS): = µ 2e(l) R { k( x (i), x (l) } . (21) l=1 Hence (20) and (19) provide the same learning process, since in both cases the kernel is real. In fact, they yield the same formula with k = k R. Next, we show that algorithms CKLMS1, CKLMS2 [6], and ACKLMS [14] are particular cases of our proposed gCKLMS algorithm in (17). They yield a subset of the cases the gCKLMS algorithm presented in this paper can represent. First, these approaches do not have a pseudo-kernel term, therefore they provide simplified limited versions and hence a reduction on the flexibility the general algorithm provides. It is easy to check that if we set the pseudo-kernel equal to zero in i − 1 i − 1 (17) the gCKLMS reduces to the CKLMS2 in (18). However, y ˆ( i ) = µ Σ e ( l ) k ( x ( i ) , x ( l )) + µ Σ e ∗ ( l ) k ˜ ( x ( i ) , x ( l )) . to have k ˜ ( x ( i ) , x ( l )) = 0 in (16) the following conditions l=1 l=1 (17) must be satisfied: k rr ( x (i), x (l)) = k jj ( x (i), x (l)), V. CONNECTION WITH OTHER ALGORITHMS In [6], two realizations of the complex-valued KLMS (CKLMS) algorithm were developed by following two methodologies. The first approach is based on using a complex-valued kernel for a complex RKHS through the associated feature map. In this approach, denoted in [6] as CKLMS2, the output yields: i − 1 y ˆ( i ) = µ e ( l ) k ( x ( i ) , x ( l )) . (18) l=1 The second alternative is the complexification of real RKHSs. In this approach, the space of complex-valued functions f ( x ) = f 1 ( x ) + j f 2 ( x ) is defined from functions f 1 ( x ) and f 2 ( x ) that are in a RKHS of real functions with real kernel k R. Then, the complexified real kernel trick allows to construct a kernel adaptive algorithm denoted in [6] as CKLMS1: i − 1 y ˆ( i ) = µ 2e ( l ) k R ( x ( i ) , x ( l )) . (19) l=1 Notice that the kernel used in this CKLMS1 algorithm is a real-valued function. In [14] the framework of [6] is applied to develop widely linear adaptive filters in complex RKHS. Two realizations of the augmented CKLMS (ACKLMS) were proposed. First, by using the complexification approach they obtain exactly the same formula (19) for the CKLMS1 algorithm (except for a rescaling) [14]. On the other hand, when a pure complexvalued kernel is used, the ACKLMS algorithm yields i − 1 y ˆ( i ) = µ ( e ( l ) k ( x ( i ) , x ( l )) + e ( l ) k ∗ ( x ( i ) , x ( l ))) . (20) l=1 k jr ( x (i), x (l)) = − krj( x (i), x (l))), (22) and the kernel in (15) for the CKLMS2 algorithm yields k( x (i), x (l)) = 2krr( x (i), x (l)) − j 2krj( x (i), x (l)). (23) Second, if in addition to k ˜ ( x ( i ) , x ( l )) = 0 we now set k rj ( x (i), x (l)) = 0 , then the kernel in (23) becomes a realvalued function k( x (i), x (l)) = 2krr( x (i), x (l)) , and the gCKLMS simplifies to the CKLMS1 in (19) or the ACKLMS in (21). A. Kernel design The conditions that the algorithms impose on the kernel and pseudo-kernel terms must be carefully analyzed in order to choose the best algorithm and kernels for a given learning problem. The kernel in an RKHS learning algorithm encodes our assumptions about the function that is being learned [5] and provides a measure of similarity between the inputs. In [11] the kernel and pseudo-kernel in (15)-(16) are analyzed, and several remarks are provided to help to design them and to decide when they should be real or complex-valued. We use that analysis here to understand the implications of the conditions that each algorithm imposes. We start with the conditions imposed when the pseudokernel is null, i.e., the conditions in (22) that yield the complex-valued kernel in (23). For any two inputs x and x ′, the first condition k rr ( x , x ′ ) = k jj ( x , x ′ ) implies that the same measure of similarity must be used with the real and the imaginary parts of the function [11]. Hence, if we impose a null pseudo-kernel, we cannot use a kernel for the real part, krr(x, x′), and another different design for the imaginary part, kjj( x , x ′ ) . The second condition is kjr( x , x ′ ) = − krj( x , x ′ ) . k ˜ ∗ ( x ( i ) , x ( l )) k ∗ ( x ( i ) , x ( l )) = , (14) 5 − R G R G r j || || ||·|| − R G −|| − || − — − G C G TABLE I CONDITIONS IMPOSED ON THE KERNEL (15) AND PSEUDO-KERNEL (16) BY THE ALGORITHMS Algorithm Kernel Pseudo-kernel Conditions gCKLMS k ( x , x ′ ) = krr ( x , x ′ ) + kjj ( x , x ′ ) + j kjr ( x , x ′ ) − krj ( x , x ′ ) CKLMS2 k ( x , x ′ ) = 2krr ( x , x ′ ) − j2krj ( x , x ′ ) ACKLMS/CKLMS1 k ( x , x ′ ) = 2krr ( x , x ′ ) ∈ R k ˜ ( x , x ′ ) = krr ( x , x ′ ) − kjj ( x , x ′ ) + j kjr ( x , x ′ ) + krj ( x , x ′ ) krj ( x , x ′ ) = −krj ( x ′ , x ) k ˜ ( x , x ′ ) = 0 krr ( x , x ′ ) = kjj ( x , x ′ ) , kjr ( x , x ′ ) = −krj ( x , x ′ ) k ˜ ( x , x ′ ) = 0 krr ( x , x ′ ) = kjj ( x , x ′ ) , kjr ( x , x ′ ) = −krj ( x , x ′ ) = 0 But we also have k jr ( x , x ′ ) = k rj ( x ′ , x ) , because the matrixvalued kernel K(x, x′) in (11) must be positive semi-definite (property (c) in Section II). Therefore, k rj ( x , x ′ ) = k rj ( x ′ , x ) and k rj ( x , x ) = k jr ( x , x ) = 0 . This imposes a skew-symmetry in the measure of similarity between the real and the imaginary parts of the function. As an example, the complex Gaussian kernel proposed in [6] for the CKLMS2 algorithm: We conclude that algorithms CKLMS1, CKLMS2 [6], iCKLMS [19] and ACKLMS [14] cannot represent any possible complex-valued function, and yield a subset of the cases that the gCKLMS algorithm proposed in this paper can represent. This is the motivation to name our proposal ‘generalized’ CKLMS. The gCKLMS, with the kernel and the pseudo-kernel terms, provides more flexibility to model the learning problem by means of the four real-valued functions k C ( x , x ′ ) = exp − ( x − x ′∗ ) T ( x − x ′∗ )/γ 2 , (24) k rr , k jj, k rj and kjr. Hence, the gCKLMS will improve results if follows the form given in (23) and fulfils the conditions in (22), i.e., the symmetries that yield a null pseudo-kernel. However, this kernel measures similarities between the real parts of the inputs with || x r − x r ′ || 2 , while for the imaginary ones it uses xj +xj′ 2, where is the l2-norm. Also, it is not stationary, has an oscillatory behavior, and the exponent in the kernel may easily grow large and positive [24]. This might cause numerical problems and, as we show later in the experiments, it does not yield the best performance. As another example, in [19] it is proposed another complexvalued kernel for the CKLMS2 algorithm, the so-called independent kernel: k ind ( x , x ′ ) = κ ( x r , x ′ ) + κ x j , x ′ + j κ R G x r, x ′ j − κ R G ( x j, x ′ r ) , (25) where x = x r + j x j and κ R G ( z , z ′ ) = exp z z ′ 2 /γ 2 is the well-known real Gaussian kernel of real inputs z and z ′. The CKLMS2 algorithm with this kernel was named the independent CKLMS estimate (iCKLMS). This independent kernel mimics the structure in (15), but it uses the same real function κ RG for all the terms in (15), and the inputs to this function are not the complex-valued inputs x, but their real or imaginary parts. Also, for the iCKLMS estimate the pseudokernel is null, hence the independent kernel has a skewsymmetric imaginary part, i.e., κ RG ( x r , x j′ ) − κ RG ( x j , x ′r ) = (κ R G ( x ′ r , x j) κ R G ( x j′ , x r)) . When the pseudo-kernel is null, the skew-symmetry k rj ( x , x ′ ) = k rj ( x , x ′ ) in the imaginary part of the kernels used by the CKLMS2 or the iCKLSM algorithms may not be a property satisfied by many to-be-learned functions and, in such a case, enforcing a complex-valued kernel can be counterproductive. Algorithms CKLMS1 [6] and ACKLMS [14] avoid this problem by adding another condition: k rj ( x (i), x (l)) = 0 . This means that these algorithms use a real-valued kernel of the form k( x , x ′ ) = 2k rr ( x , x ′ ) . The condition k rj ( x (i), x (l)) = 0 implies that the real and the imaginary parts are not related and that one of them does not provide information to learn the other [11]. In Table I we summarize, for every algorithm, the conditions imposed on the kernel and pseudo-kernel terms. problem. Some interesting scenarios are as follows. If the real and imaginary parts of the output are not independent, and also they have different properties in terms of similarity, then the gCKLMS with a kernel and a pseudo-kernel is needed. If the real and imaginary parts of the output are independent, k rj = k jr = 0 , but they still have different properties in terms of similarity, k rr = k jj, then kernel and pseudo-kernel are needed. Finally, when the real and imaginary parts of the output are independent, and they both have the same properties in terms of similarity, the pseudo-kernel is null and a real-valued kernel should be used. In general, setting the pseudo-kernel to zero and enforcing a complex-valued kernel is counterproductive unless you identify, for the particular problem at hand, a skewsymmetry of the kind krj( x , x ′ ) = krj( x , x ′ ) . We end this discussion about the kernels by bringing here a suitable real-valued function for krr, kjj, krj and kjr proposed in [11]. This is the adaptation to complex-valued inputs of the real-valued Gaussian kernel: k G ( x , x ′ ) = exp − ( x − x ′ ) H ( x − x ′ )/γ 2 , (26) where γ is the kernel parameter. This real function provides a measure of similarity between the complex-valued inputs that is simple but effective for complex-valued signals: inputs closer to other input in the complex field are considered more similar than inputs that are further away [11]. We will use it in our experiments. For a further analysis about the selection of suitable kernels for complex-valued applications see [11], [24]. VI. EXPERIMENTS We consider two experiments where we compare the performance of our proposal the gCKLMS in (17), versus the CKLMS2 in (18) [6], the iCKLMS [19], and the ACKLMS algorithm in (21) [14]. In the first experiment, we reproduce the nonlinear channel equalization task in [14]. In this experiment, the complexvalued signals have independent real and imaginary parts, and they are better represented with different kernels. We show that in such a case the best choice is a real kernel and a real pseudo-kernel. the conditions described above are not suitable for our learning 6 *EGJak i*EGJa A*EGJa ab iM (R9) A*EGJa riih F2`M2H UkeV ;*EGJa *EGJak i*EGJa A*EGJa ab iM (R9) A*EGJa riih F2`M2H UkeV ;*EGJa *EGJak i*EGJa A*EGJa ab iM (R9) A*EGJa riih F2`M2H UkeV ;*EGJa Ap2`a;2/ Ja1 U/BV − 4 − 6 − 8 − 10 −120 1000 2000 3000 4000 5000 n Fig. 1. MSE in dB versus the number of input samples for the soft nonlinear channel with the circular input case for the CKLMS2 ( γ CG = 10), the iCKLMS ( γ RG = 5), the ACKLMS as in [14] ( γ CG = 10), the ACKLMS with kernel (26) ( γ r = 5 ), and the gCKLMS ( γ r = 6 . 5 and γ j = 5 . 5 ). In the second experiment, we propose learning a filtered two-dimensional random process. At the output of the filter, the real and imaginary parts are not independent, and we show that we can use the imaginary part of the pseudo-kernel to improve the performance. − 2 − 4 − 6 − 8 − 10 − 12 −140 1000 2000 3000 4000 5000 n Fig. 2. MSE in dB versus the number of input samples for the soft nonlinear channel with the noncircular input case for the CKLMS2 ( γ CG = 10), the iCKLMS ( γ RG = 5), the ACKLMS as in [14] ( γ CG = 10), the ACKLMS with kernel (26) ( γ r = 5 ), and the gCKLMS ( γ r = 6 . 5 and γ j = 5 . 5 ). − 6 As in [14], we use the complex Gaussian kernel kCG(x, x′) − 8 in (24) for both the CKLMS2 and the ACKLMS. In fact, for the ACKLMS the real part of this kernel is used, as was shown in (21). We use the code available in [25] to run the algorithms. For the iCKLMS, the CKLMS2 in (18) with the independent kernel (25) is used. For our proposed gCKLMS we use the general kernel and pseudo-kernel in (15)-(16). For k rr ( x , x ′ ) , k jj ( x , x ′ ) , k jr ( x , x ′ ) and krj(x, x′) we propose to use the real-valued Gaussian kernel k G ( x , x ′ ) in (26) with parameter γ = γ r for k rr, γ = γ j for kjj , γ = γrj for krj , and γ = γjr for kjr , respectively. The kernel and pseudo-kernel can be simplified if the signals meet any of the conditions discussed in Section V-A. For example, if the real and imaginary parts of the signals are independent we can set kjr( x , x ′ ) = krj( x , x ′ ) = 0 and the kernel and pseudo-kernel are real-valued: − 10 − 12 0 1000 2000 3000 4000 5000 n Fig. 3. MSE in dB versus the number of input samples for the strong nonlinear channel with the circular input case for the CKLMS2 ( γ CG = 15), the iCKLMS ( γ RG = 5), the ACKLMS as in [14] ( γ CG = 15), the ACKLMS with kernel (26) ( γ r = 5), and the gCKLMS ( γ r = 5 and γ j = 3). − 4 k( x , x ′ ) = k rr 6 ( x , x ′ ) + kjj( x , x ′ ), (27) k ˜ ( x , x ′ ) = k rr ( x , x ′ ) − k jj ( x , x ′ ) . (28) We use this simplification in the first experiment. Notice that if we also assume that the real and imaginary parts of the output use the same kernel, krr = kjj , then we should set γr = γj and the pseudo-kernel term in (28) cancels. In such a case, as explained in Section V, the gCKLMS approach simplifies to the ACKLMS with real-valued kernel k( x , x ′ ) = 2k rr ( x , x ′ ) , where k rr ( x , x ′ ) is as in (26) with γ = γ r. We will refer to this case as ACKLMS with kernel (26) in the experiments. A. Nonlinear channel equalization We face the problem of nonlinear channel equalization, as in [6] and [14], for ease of comparison and continuity. The channel consists of a linear filter and a memoryless nonlinearity. The two nonlinear channels in [14] have been considered − 8 − 10 − 12 − 14 −160 1000 2000 3000 4000 5000 n Fig. 4. MSE in dB versus the number of input samples for the strong nonlinear channel with the noncircular input case for the CKLMS2 (γCG = 15), the iCKLMS ( γ RG = 5), the ACKLMS as in [14] ( γ CG = 15), the ACKLMS with kernel (26) ( γ r = 5 ), and the gCKLMS ( γ r = 5 and γ j = 3 ). here. The first channel is the soft nonlinear channel, with linear *EGJak i*EGJa A*EGJa ab iM (R9) A*EGJa riih F2`M2H UkeV ;*EGJa Ap2`a;2/ Ja1 U/BV Ap2`a;2/ Ja1 U/BV Ap2`a;2/ Ja1 U/BV − 7 √ − {− } filter t(n) = ( − 0.9 + 0.8 j ) · s(n) + (0.6 − 0.7 j ) · s(n − 1), followed by the nonlinearity q(n) = t(n) + (0.1 + 0.15 j ) · t 2 (n) + (0.06 + 0.05 j ) · t 3 (n). The second one is the strong nonlinear channel, with linear filter t(n) =( − 0.9 + 0.8 j ) · s(n) + (0.6 − 0.7 j ) · s(n − 1) + ( − 0.4 + 0.3 j ) · s(n − 2) + (0.3 − 0.2 j ) · s(n − 3) + ( − 0.1 − 0.2 j ) · s(n − 4), and nonlinearity q(n) = t(n) + (0.2 + 0.25 j ) · t 2 (n) + (0.08 + 0.09 j ) · t 3 (n). At the receiver, the signal q(n) is corrupted by additive white circular Gaussian noise with an SNR of 15 dB to yield the received signal r(n). The inputs to the equalizer are the other algorithms, is the key to obtain a better estimation. In this experiment, the gain in MSE is small, because krr(x, x′) and kjj(x, x′) are very similar. That is the reason why the ACKLMS with kernel (26), i.e., setting γ r = γ j, performs well and close to the general case. In any case, it can be observed that to achieve a given error, the faster convergence of the gCKLMS allows saving 10%-30% of the samples and time. The dictionary size after sparsification and the running times of the tested algorithms are in Table II and III 1 . With the complex Gaussian kernel, both the ACKLMS in [14] and the CKLM2 in [6] perform poorly compared to the gCKLMS. Therefore, the experiments show that the complex Gaussian kernel is not the best choice for this equalization task and, as it is shown in Fig. 2, sometimes yields undesired spikes in the learning curves. TABLE II DICTIONARY SIZE AFTER SPARSIFICATION AND RUNNING TIMES (IN SECONDS) FOR THE SOFT NONLINEAR CHANNEL AND GAUSSIAN DISTRIBUTED INPUTS sets of samples x = [r(n + D), r(n + D − 1), · · · , r(n + D L + 1)] T, where L > 0 is the filter length and D is the equalization time delay. Here we set L = 5 and D = 2 , as in [14]. The goal is to estimate the original input signal s(n) . 1) Gaussian distributed inputs: We first set the input signals as in [14]: s(n) = 0.7( 1 − ρ 2 · X(n) + j ρ · Y (n)) , where X ( n ) and Y ( n √ ) are independent Gaussian random Circular Input Noncircular Input variables, with ρ = 1/ 2 for circular signals, and ρ = 0.1 for noncircular signals. The real and imaginary parts of the signals are independent and, therefore, we can set k jr ( x , x ′ ) = k rj ( x , x ′ ) = 0 and use the real-valued kernel and pseudo-kernel terms in (27)-(28) for TABLE III DICTIONARY SIZE AFTER SPARSIFICATION AND RUNNING TIMES (IN SECONDS) FOR THE STRONG NONLINEAR CHANNEL AND GAUSSIAN DISTRIBUTED INPUTS Figs. 1 and 2 show the averaged mean square errors (MSE) for the soft nonlinear channel. The circular input case is shown in Fig. 1, and the noncircular input case is shown in Fig. 2. Since we are reproducing here the experiment in [14], for the CKLMS2 and the ACKLMS algorithms we set γ CG = 10 and µ = 1/8 , i.e., the same values used in [14]. For the iCKLMS we set γ RG = 5 and µ = 1/8 . For the gCKLMS algorithm we set γ r = 6.5 and γ j = 5.5 , and µ = 1/7 . For the ACKLMS with kernel (26) we set γ r = 5 and µ = 1/10 . Figs. 3 and 4 include the MSE for the strong nonlinear channel and the circular input and noncircular input cases, respectively. Again, since we are reproducing the experiment in [14], for the CKLMS2 and the ACKLMS algorithms we set γ CG = 15 and µ = 1/6 , i.e., the same values used in [14]. For the iCKLMS we set γ RG = 5 and µ = 1/8 . For the gCKLMS algorithm we set γ r = 5 and γ j = 3 , and µ = 1/7 . For the ACKLMS with kernel (26) we set γ r = 5 and µ = 1/10 . In all the examples, the proposed gCKLMS outperforms the other algorithms. The main advantage of the gCKLMS is that by introducing a pseudo-kernel we can use a different kernel for the real and the imaginary parts, k rr ( x , x ′ ) and k jj ( x , x ′ ) . This extra degree of freedom, which is not present in the 2) Unbalanced digital modulated signals: In digital communications inputs are discrete. For discrete and unbalanced digital modulated signals, the difference between krr(x, x′) and kjj(x, x′) is greater and the proposed gCKLMS algorithm is a good choice versus the previous proposals, with null pseudokernels. To illustrate this point, we propose to repeat here the equalization experiment for the soft nonlinear channel, where the input signals are now s(n) = 0.2X(n) + j 0.1Y (n) , where X(n) and Y (n) are independent binary 1, +1 data streams. For the proposed gCKLMS algorithm we use again the realvalued Gaussian kernel (26) with parameters γ r = 0.59 for k rr ( x , x ′ ) and γ j = 1.63 for k jj ( x , x ′ ) , respectively. For the ACKLMS with kernel (26) we set γ r = 1.52 . The learning parameter is set to µ = 0.5 for both approaches. We generate 100 independent test trials with a set of 10000 samples to test the algorithms. The mean square errors (MSE) of the estimation are compared in Fig. 5 versus the number of input samples. It can be observed that the proposed gCKLMS 1Matlab 2018a on a computer with Windows Server 2016, 64 GB RAM and Intel Xeon 2.80 GHz Processor Algorithm Dict. size Run time (s) Dict. size Run time (s) gCKLMS 2966 4.6181 1866 3.5118 CKLMS2 3291 4.6314 2692 4.1555 ACKLMS [14] 3173 4.4803 2102 3.7701 ACKLMS, kernel (26) 3038 3.8185 1926 3.2533 iCKLMS [19] 3419 5.5193 2692 3.3111 our proposed gCKLMS. Algorithm Circul Dict. size ar Input Run time (s) Noncircular Input Dict. size Run time (s) Experiments were conducted on 100 independent sets of gCKLMS 2565 5.1858 1439 3.0639 5000 samples of the input signal. For all the approaches, the CKLMS2 ACKLMS [14] 3222 3259 6.3349 5.3047 2565 2220 3.5103 3.4936 novelty criterion [26], [27], is used for sparsification with δ1 = ACKLMS, kernel (26) 2698 4.7781 1528 2.6946 0.15 and δ 2 = 0.2 , as in [14]. iCKLMS [19] 2958 4.7952 2312 4.2762 8 { } ∈ − ∈ − ∈ − ∈ − −| | −| | · { } — − 0 − 5 − 10 − 15 0 2000 4000 6000 8000 10000 n Fig. 6 shows the estimation MSE of the imaginary part of the signals, while Fig. 7 shows the estimation MSE of the real part. In this experiment the real and imaginary parts require a different kernel to be accurately learnt. However, the ACKLMS algorithm uses the same kernel for both parts. The parameter value γ r with the best performance to learn the imaginary part of the output in Fig. 6 yields the worst estimation of the real part in Fig. 7. And vice versa, the best parameter value to learn the real part of the output in Fig. 7 provides the worst performance in Fig. 6. Remarkably, the estimation with the proposed gCKLMS is always low for both imaginary and real parts, as it allows to set different values for k rr ( x , x ′ ) and k jj ( x , x ′ ) . Fig. 5. MSE in dB versus the number of input samples for the ACKLMS with kernel (26) ( γ r = {0 . 5 , 1 , 1 . 52}) and the gCKLMS ( γ r = 0 . 59, γ j = 1 . 63). 0 − 10 − 20 0 2000 4000 6000 8000 10000 n Fig. 6. MSE in dB for the imaginary part versus the number of input samples for the ACKLMS with kernel (26) (γr = 0.5, 1, 1.52 ) and the gCKLMS ( γ r = 0 . 59, γ j = 1 . 63). − 12 B. A random process filtered In this experiment, we show the performance of the proposed gCKLMS when the signals do not have independent real and imaginary parts. We generate a complex-valued signal with correlated real and imaginary parts by filtering a real-valued random process with a complex-valued filter. We define the complex-valued filter h(x) = α 2 exp( x 2 /3) + j exp( x 2 /0.5 , where x = xr + j xj , with xr [ 5, 5] and xj [ 5, 5], and α = 0.0228 to ensure unit norm. Then we define a real Gaussian process s(xr, xj) with zero mean and unit variance, and we pass this process through the filter. We show in Figs. 8 and 9 the real and imaginary parts of one sample of the filtered process in xr [ 5, 5] and xj [ 5, 5]. At the output of the filter we get a Gaussian process. There is a known relationship between the variance of a Gaussian process and the kernel, and between the pseudo-covariance of the Gaussian process and the pseudo-kernel [8]. Therefore, the covariance and pseudo-covariance of the process help in the design of suitable kernels. It can be shown that the covariance of the process at the output of the filter is real-valued with two additive terms. Hence, we can conclude that for the gCKLMS we could use a kernel with the following terms: − 14 ′ ′ ′ − 16 − 18 0 2000 4000 6000 8000 10000 n Fig. 7. MSE in dB for the real part versus the number of input samples for the ACKLMS with kernel (26) (γr = 0.5, 1, 1.52 ) and the gCKLMS ( γ r = 0 . 59, γ j = 1 . 63). outperforms the ACKLMS with kernel (26), i.e., the case with null pseudo-kernel. Again, the key to obtaining a better estimation with the gCKLMS in this experiment is the possibility to define a different kernel for the real part krr(x, x′) and the imaginary part kjj(x, x′). Figs. 6 and 7 are included to highlight this. k( x , x ) = krr( x , x ) + kjj( x , x ). (29) Also, it can be found that the pseudo-covariance of the process is complex-valued, and this is an indication that there exists a correlation between the real and the imaginary parts of the filtered process [1]. Thus, we can conclude that for the gCKLMS we could use a pseudo-kernel with the following terms: k ˜ ( x , x ′ ) = k rr ( x , x ′ ) − k jj ( x , x ′ ) + 2 j k jr ( x , x ′ ) . (30) All the terms in the covariance and pseudo-covariance are exponentials of the form exp( d ∗ x d x /γ ) , where d x = x x ′ [8], having different γ values and scalings. Accordingly, we propose to use the real-valued Gaussian kernel kG(x, x′) in (26) for k rr ( x , x ′ ) , k jj ( x , x ′ ) and k jr ( x , x ′ ) , with different scaling and kernel parameters. Further details about the construction of kernels based on covariances can be found in [8]. In the gCKLMS algorithm, the real and imaginary parts of the error e provide two degrees of freedom to adjust the amplitudes of the kernel terms as the learning process A*EGJa riih F2`M2H UkeV, γ ` = 1.52 A*EGJa riih F2`M2H UkeV, γ ` = 1 A*EGJa riih F2`M2H UkeV, γ ` = 0.5 ;*EGJa A*EGJa riih F2`M2H UkeV, γ ` = 1.52 A*EGJa riih F2`M2H UkeV, γ ` = 1 A*EGJa riih F2`M2H UkeV, γ ` = 0.5 ;*EGJa A*EGJa riih F2`M2H UkeV, γ ` = 1.52 A*EGJa riih F2`M2H UkeV, γ ` = 1 A*EGJa riih F2`M2H UkeV, γ ` = 0.5 ;*EGJa Ja1 U/BV Ja1 U/BV Ja1 U/BV