Application of non-parametric regression in engineering optimization

Abstract—Non-parametric regression models (e.g. kernel smoothing techniques, support vector regression model) have been widely used in statistics and recently in econometrics and engineering as well. Regression functions can be effective tools in the solution of engineering optimization problems. In the investigation of complex technical systems the class of the function describing the connection between the input and output data is generally unknown, moreover the “classical” least square fitting method is not flexible enough to provide an effective regression function in the case of higher dimensional optimization problems. In this paper regression functions obtained in different models are compared graphically in one dimensional case. Calculations were made using Maple, R, and MS Excel software. Keywords—kernel function, regression I. INTRODUCTION HE smoothing techniques and the non-parametric approach have a long tradition in empirical analysis. The intensive interest in smoothing over the previous decades had two reasons: “statisticians realized that pure parametric thinking in curve estimations often does not meet the need for flexibility in data analysis and the development of hardware created the demand for theory of now computable nonparametric estimates.” [1] Kernel smoothing is an effective regression technique for engineering optimization as well. It has the advantage that the function defining the non-linear relation between the input and output data does not need to be given in explicit form. As an example the shape optimization problem of rubber bumpers can be mentioned that was investigated by the second author in his doctoral thesis [7]. To present the flexibility of the kernel smoothing and the support vector regression we give the regression function for certain input data (one dimensional case). II. KERNEL FUNCTIONS Let us consider a data set (frequently called training date set)      1N PP11 Rd,t,...,d,t   , (1) where i t are (possibly one dimensional) input vectors, i d are associated targets ( R is the set of reals). The goal is to fit a function RR:f N which approximates the relation between the data sets. Any regression algorithm has a loss function    d,tfL which describes how the estimated function deviated from the true one. In this note we refer to the following two types of loss function: the quadratic loss function (least square fitting)       2 iiii dtfd,tfL  (2) and the  -insensitive (Vapnik’s) loss function               otherwisedtf dtfif0 d,tfL ii ii ii , (3) where  is a fixed positive parameter. It is well-know that in the linear model (with the quadratic loss function)   mindbtw w P 1i 2 ii    (4) the regression function can be expressed as a linear combination of so-called kernel functions   ii txxk  (5) in the form        P 1i ii opt bxkbxwxf , (6) where  denotes the inner product [5]. In non-linear models the situation is similar: applying a suitable transformation MN RR:   the problem will be linear in M R and the regression function is a linear combination of kernel functions       ii txxk  (7) [3], [4] in the form          P 1i ii opt bxkbxwxf . (8) Since function  and, consequently, functions i k are generally unknown in practice, one of the recommended APPLICATION OF NON-PARAMETRIC REGRESSION IN ENGINEERING OPTIMIZATION Imre KOCSIS1, Tamas MANKOVITS2 1University of Debrecen Faculty of Engineering, [email protected]deb.hu 2 University of Debrecen Faculty of Engineering, tamas.man[email protected]eb.hu T ANNALS OF THE ORADEA UNIVERSITY Fascicle of Management and Technological Engineering ISSUE #1, MAY 2013, http://www.imtuoradea.ro/auo.fmte/ 159 kernel functions is chosen in the model to build f . The most used kernel in technical literature is the radial base Gaussian kernel function   2 i tx iexk   , (9) where  is a parameter determining the shape of the kernel function. The role of  is presented graphically in section III. III. APPLICATION OF GAUSSIAN KERNEL FUNCTION As an input data set consider the set of the following points {(ti,di)|i=1,..,P}={(1,16),(2,19),(3,9),(4,16),(5,25), (6,10),(7,5),(8,4),(9,24),(10,20)} (Fig.5.) and solve the quadratic optimization problem     mind)t(f bebxk)x(f b, P 1i 2 ii P 1i tx i P 1i ii 2 i          (10) for P P1 R),...,(  and Rb , where  is a fixed positive number. The following figures show graphically the regression functions obtained for certain values of  . (Calculations and plotting were carried out using Maple). 05.0  1.0  Fig. 1. The regression functions (solutions of the optimization problem (10)) when  =0.05 and  =0.1 15.0  1  Fig. 2. The regression functions (solutions of the optimization problem (10)) when  =0.15 and  =1 2  3  Fig. 3. The regression functions (solutions of the optimization problem (10)) when  =2 and  =3 It can be seen that the smaller values of  result flatter regression functions (see e.g. Fig.1.) but the deviation of these “flat” functions at the learning points can be relatively high. Large values of  lead to regression functions varying too fast and the shape of the regression functions do not seem to be reasonable (see e.g. Fig.4.). Analyzing the pictures it can be seen that the values 2..1  mean an acceptable compromise between the flatness of the function and the accuracy at the learning points (Fig.2., Fig.3.). ANNALS OF THE ORADEA UNIVERSITY Fascicle of Management and Technological Engineering ISSUE #1, MAY 2013, http://www.imtuoradea.ro/auo.fmte/ 160 5  10  Fig. 4. The regression functions (solutions of the optimization problem (10)) when  =5 and  =6 IV. SUPPORT VECTOR REGRESSION MODEL Application of the  -insensitive loss function (3) leads to a quadratic convex optimization problem and the regression function can be expressed by kernel functions. In the linear model the regression function is sought in the form   Rb,Rw,bxwxf N . (11) The flatness of f in this case means that one seeks a small w , that is, the norm w is to be minimized. The related convex optimization problem is the following:     .P,...,1i, dbtw btwd tosubject w 2 1 minimize ii ii 2       (12) Inequalities in (12) mean that the regression function is required to approximate the pairs   ii d,t with  precision. Sometimes the convex optimization problem is infeasible with the given constraints and some “errors” must be allowed. For this purpose so-called slack variables was introduced by Vapnik in [2]. The modified convex optimization problem with slack variables i  and ' i  is the following       .P,...,1i, 0',0 'dbtw btwd tosubject 'Cw 2 1 minimize ii iii iii P 1i ii 2             (13) The positive constant C determines the trade-off between the flatness of f and the amount up to which deviations larger than  are tolerated. According to the standard dualization method [6] we introduce the Lagrange function                  P 1i iii P 1i iii P 1i iiii P 1i ii 2 iiii )bxw(d'' d)bxw('' 'Cw 2 1 )',,',,b,w(L (14) where 0',,', iiii   . The dual optimization problem is the following:           ].C,0[', and 0' tosubject t'' tt'' 2 1 maximize ii P 1i ii i P 1i ii P 1i ii jijj P 1j,i ii          (15) Solving the dual problem we get     btx'xf i P 1i ii    , (16) that is, the solution is a linear combination of kernel functions. This form of f says that the explicit form of w does not need to be computed. Furthermore it can be proved that for the learning points inside the  -tube 0' ii   , that is, f is determined by the learning points having nonvanishing coefficients. These pairs are called support vectors. As it mentioned before, in non-linear model the application of a suitable transformation function MN RR:   leads to a linear problem. It can be proved that in this case regression function is a linear combination of kernel functions       ii txxk   . The convex optimization problem is the following         .P,...,1i, 0',0 'dbtw btwd tosubject 'Cw 2 1 minimize ii iii iii P 1i ii 2             (17) The dual problem is               ].C,0[', and 0' tosubject t'' tt'' 2 1 maximize ii P 1i ii i P 1i ii P 1i ii jijj P 1j,i ii          (18) Solving the dual problem we get         btx'xf i P 1i ii    , (19) that is, the solution is a linear combination of kernel ANNALS OF THE ORADEA UNIVERSITY Fascicle of Management and Technological Engineering ISSUE #1, MAY 2013, http://www.imtuoradea.ro/auo.fmte/ 161 functions. Using the SVR package of the R software we present the role of parameters  ,  and C in the SVR model. We consider the same input data set as in section II: {(ti,di)|i=1,..,P}={(1,16),(2,19),(3,9),(4,16),(5,25),(6,10), (7,5),(8,4),(9,24),(10,20)} (Fig.5.) and choose the Gaussian kernel function (9). (Plotting was carried out using MS Excel.) Fig. 5. The set of input data Fig.6. shows the role of the “penalty” parameter C . Application of a higher value of C results a regression function with small deviation even in outlying learning points, while low value of C gives a flat function with large deviation at certain points. Fig. 6. The role of the “penalty” parameter C (C=0.1,1,10,  =5,  =0.05) Fig. 7. The role of the parameter  (  =2,5,8, C=10,  =0.05) Fig. 8. The role of the parameter  (  =0.05,0.1,0.3,0.5,  =5, C=10) Fig.7. shows the role of the parameter  (it was described also in section II). Fig.8. shows the role of the parameter  . The learning points are possibly (depending on the value of the slack variables) in an  - tube or near the  -tube. Examples presented in this note show the flexibility of the regression methods using kernel functions. In engineering applications this flexibility can have an important role when the model has to be adjusted to the special characteristics of the input data set or to the requirements related to the regression function. ACKNOWLEDGMENT The described work was carried out as part of the TÁMOP-4.2.2/B-10/1-2010-0008 and TÁMOP-4.2.2.A11/1/KONV-2012-0041 projects in the framework of the New Hungarian Development Plan. The realization of this project is supported by the European Union, cofinanced by the European Social Fund. REFERENCES [1] W. Hardle, “Applied non-parametric regression”, HumboltUniversität, Berlin, 1994 [2] C. Cortes and V. Vapnik, “Support vector networks”, Machine Learning, Vol. 20, pp. 273-297, 1995. [3] B. Schölkopf, and A. J. Smola, “Learning with Kernels”, MIT Press, 2000 [4] A. Farag and R. M. Mohamed, “Regression Using Support Vector Machines: Basic Foundations”, University of Louisville, 2004 [5] S. Haykin, “Neural Networks and Learning Machines”, Prentice Hall, 2009 [6] S. Boyd and L. Vandenberghe, “Convex Optimization”, Cambridge University Press, 2009 [7] T. Mankovits, “Shape optimization of rubber parts”, PhD Thesis, University of Miskolc, 2013 ANNALS OF THE ORADEA UNIVERSITY Fascicle of Management and Technological Engineering ISSUE #1, MAY 2013, http://www.imtuoradea.ro/auo.fmte/ 162