Dimensionality reduction, and function approximation of poly(lactic-co-glycolic acid) micro- and nanoparticle dissolution rate

© 2015 Ojha et al. This work is published by Dove Medical Press Limited, and licensed under Creative Commons Attribution – Non Commercial (unported, v3.0) License. The full terms of the License are available at http://creativecommons.org/licenses/by-nc/3.0/. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. Permissions beyond the scope of the License are administered by Dove Medical Press Limited. Information on how to request permission may be found at: http://www.dovepress.com/permissions.php International Journal of Nanomedicine 2015:10 1119–1129 International Journal of Nanomedicine Dovepress submit your manuscript | www.dovepress.com Dovepress 1119 OrIgINal research open access to scientific and medical research Open access Full Text article http://dx.doi.org/10.2147/IJN.S71847 Dimensionality reduction, and function approximation of poly(lactic-co-glycolic acid) microand nanoparticle dissolution rate Varun Kumar Ojha1,2 Konrad Jackowski3 ajith abraham1,4 Václav snášel1,2 1IT4Innovations, VŠB – Technical University of Ostrava, Ostrava, czech republic; 2Department of computer science, VŠB – Technical University of Ostrava, Ostrava, czech republic; 3Department of systems and Computer Networks, Wrocław University of Technology, Wrocław, Poland; 4Machine Intelligence research labs, auburn, Wa, Usa Abstract: Prediction of poly(lactic-co-glycolic acid) (PLGA) microand nanoparticles’ dissolution rates plays a significant role in pharmaceutical and medical industries. The prediction of PLGA dissolution rate is crucial for drug manufacturing. Therefore, a model that predicts the PLGA dissolution rate could be beneficial. PLGA dissolution is influenced by numerous factors (features), and counting the known features leads to a dataset with 300 features. This large number of features and high redundancy within the dataset makes the prediction task very difficult and inaccurate. In this study, dimensionality reduction techniques were applied in order to simplify the task and eliminate irrelevant and redundant features. A heterogeneous pool of several regression algorithms were independently tested and evaluated. In addition, several ensemble methods were tested in order to improve the accuracy of prediction. The empirical results revealed that the proposed evolutionary weighted ensemble method offered the lowest margin of error and significantly outperformed the individual algorithms and the other ensemble techniques. Keywords: feature selection, regression models, ensemble, protein dissolution Introduction Predicting the poly(lactic-co-glycolic acid) (PLGA) microand nanoparticle dissolution profiles presents a complex and vital problem. The complexity of the problem can be understood from the fact that academic literature1–18 provides 300 potential factors that may influence the dissolution of the PLGA protein particles.19 After analyzing the collected dataset, the primary approach adopted in most research has been to reduce the dimensionality of the dataset. Dimensionality reduction techniques transform high-dimensional datasets into low-dimensional datasets, thereby improving the model’s computational speed, predictability, and generalization ability. Dimensionality reduction techniques are classified into two categories: feature selection, and feature extraction. The feature selection technique is useful when the available dataset has a large dimension and relatively few cases (samples), whereas the feature extraction technique is useful when the dataset has a large dimension and high redundancy.20 The dataset in the present research had a large dimension, and the features appeared to have high redundancy. Therefore, it was not immediately clear to us whether we should use feature selection or feature extraction. Hence, we explored both feature selection and feature extraction techniques in order to find the best possible solution. Several regression models were employed to evaluate the relationship between the obtained input variables (features) and output variable. correspondence: Varun Kumar Ojha IT4Innovations, VŠB – Technical University of Ostrava, 17 listopadu 15/2172, 708 33 Ostrava – Poruba, Ostrava, czech republic Tel +420 777 880431 email [email protected] Journal name: International Journal of Nanomedicine Article Designation: Original Research Year: 2015 Volume: 10 Running head verso: Ojha et al Running head recto: Function approximation of PLGA micro and nanoparticle dissolution rate DOI: http://dx.doi.org/10.2147/IJN.S71847 This article was published in the following Dove Press journal: International Journal of Nanomedicine 4 February 2015 Number of times this article has been viewed International Journal of Nanomedicine 2015:10 submit your manuscript | www.dovepress.com Dovepress Dovepress 1120 Ojha et al In the scope of the present study, our focus was on PLGA nanoand microsphere dissolution properties and drug release rate. Szle˛k et al21 and Fredenberg et al22 described that drug release from the PLGA matrix is mainly governed by two mechanisms: diffusion, and degradation/erosion. Several factors influencing the diffusion and degradation rates of PLGA as described by Kang et al, 23,24 Blanco and Alonso,25 and Mainardes and Evangelista26 include pore diameters, matrix active pharmaceutical ingredient (API) interactions, API–API interactions, and the composition of the formulation. Szle˛k et al21 developed a predictive model to describe the underlying relationship between those influencing factors on the drug’s release profile, and they focused on feature selection, artificial neural network, and genetic programming approaches to come up with a suitable prediction model. In the past, several mathematical models, including the Monte Carlo and cellular automata microscopic models, were proposed by Zygourakis and Markenscoff,27 and Gopferich.28 A partial differential equations model was proposed by Siepmann et al29 to address the influence of underlying PLGA properties on the drug’s release rate/protein dissolution. The highlights of the present article are as follows: • a comprehensive discussion on the drug release problem and dataset collection mechanisms; • a comprehensive discussion on various computational tools used to reduce dimensionality of dataset; • a concise discussion on the elementary regression models available in the literature; • a concise discussion on the ensemble methods used for making ensembles of the elementary regression models; • a comprehensive discussion and conclusion on the experimental results mentioned in the present article. Methodology a description of the problem PLGA microand nanoparticles could play a significant role in the medical application and toxicity evaluation of PLGAbased multi-particulate dosages.30 PLGA micro-particles are important diluents used to produce drugs in their correct dosage form. Apart from playing the role as a filler, PLGA as an excipient, and alongside pharmaceutical APIs, plays other crucial roles in various ways. It helps in the dissolution of drugs, thus increasing the absorbability and solubility of drugs.31,32 It helps in pharmaceutical manufacturing processes by improving API powders’ flow and non-stickiness. The dataset collected from various academic literature1–18 contains 300 input features categorized into four groups, including protein descriptor, plasticizer, formulation characteristics, and emulsifier. A detailed description of the dataset is given in Table 1. For example, the formulation characteristics group contains features such as PLGA-inherent viscosity, PLGA molecular weight, lactide-to-glycolide ratio, inner and outer phase polyvinyl alcohol (PVA) concentration, PVA molecular weight, inner phase volume, encapsulation rate, mean particle size, PLGA concentration, and experimental conditions (dissolution pH, the number of dissolution additives, dissolution additive concentration, and production method and dissolution time). The protein descriptor, plasticizer, and emulsifier feature groups contain 85, 98, and 101 features, respectively. The regression model sought to predict the dissolution percentage or solubility of PLGA, which is dependent on the features mentioned above. In order to avoid over-fitting, collected data were preprocessed by adding noise to them. The dataset was then normalized, in other words, scaled within the range –1.0 and 1.0. Dimensionality reduction Feature selection tools Feature selection techniques enable us to identify the most relevant input feature from the available set of input features and allows us to avoid expensive (both in time and cost) experimental examination while developing a prediction model.33 Backward feature elimination Backward feature elimination filtering starts with the maximum number of features (in this case, it starts with 300 features) and eliminates them one-by-one in an iterative manner. At each iteration, the resulting accuracy of Table 1 The Plga dataset description Sl No Group name No of features Importance 1 Protein descriptors 85 Describes the type of molecules and proteins used 2 Formulation characteristics 17 Describe the molecular properties such as molecular weight, particle size, etc 3 Plasticizer 98 Describe the properties such as fluidity of the material used 4Emulsifier 99 Describe the properties of stabilizing/increase the pharmaceutical product life 5 Time in days 1 Time taken to dissolve 6 % of molecules dissolved 1 Output Abbreviations: Plga, poly(lactic-co-glycolic acid); sI, serial; No, number. International Journal of Nanomedicine 2015:10 submit your manuscript | www.dovepress.com Dovepress Dovepress 1121 Function approximation of Plga micro and nanoparticle dissolution rate prediction is evaluated for all combinations of the remaining attributes. The subsets of attributes with the high accuracies are propagated to the next iteration. Finally, the subset with the highest degree of accuracy (the lowest root mean square error [RMSE]) is selected as the best subset. correlation-based feature selection Correlation-based feature selection assesses the value of a group of attributes that concern the individual predictive ability of each feature, as well with the possibility of repetition among the features.34 Classifier-based feature selection Classifier-based feature selection evaluates attribute subsets on training data and uses a classifier to estimate the merits of a set of attributes. A search algorithm is then applied to search for a suitable feature from among all the available feature sets. Wrapper feature selection Wrapper-based feature selection evaluates attribute sets by using a learning scheme, and then uses cross-validation (CV) to estimate the accuracy of the learning scheme for a particular set of attributes.35 A search algorithm is then applied to search for a suitable feature set from among all the available feature sets. Feature extraction When it is affordable to easily generate test features, feature extraction techniques may be useful for dimensionality reduction. A regression model with a reduced input dimension may perform as well as it can if it has a complete set of features.20 Therefore, feature extraction helps in reducing the computational overhead that may be incurred when using a complete input dimension. Principle component analysis Principle component analysis (PCA) is a linear dimensionality reduction technique that transforms correlated data into uncorrelated data in a reduced dimension by finding a linear basis of reduced dimensionality for data with maximal variance. More specifically, it transfers correlated variables into a set of linearly uncorrelated variables called principle components.36,37 Factor analysis Factor analysis (FA), as opposed to PCA, determines whether a number of features of interest are linearly related to a smaller/reduced number of newly-defined features called factors. In other words, it discovers a reduced number of relatively independent features by mapping correlated features to a small set of features known as factors.38 Independent component analysis Independent component analysis (ICA), proposed by Hyvärinen and Oja39 and Hyvärinen,40 is a linear dimension reduction technique that transforms multidimensional feature vectors into components that are statistically as independent as possible. More specifically, ICA maps the observed variables (features) to a small number of latent variables (features) that are non-Gaussian and mutually independent, and they are called the independent components of the observed data.41 Kernel Pca Kernel PCA (kPCA) is an extension of PCA that uses kernel methods. kPCA computes the principal eigenvectors of the kernel matrix, rather than those of the covariance matrix.42 Reformulating PCA in the kernel space is straightforward, since a kernel matrix is similar to the inner product of the data points in the high-dimensional space that is constructed using the kernel function. Typically, Gaussian, tangent hyperbolic, polynomial, and other functions are used for the kernel. Multidimensional scaling Multidimensional scaling (MDS) is a non-linear dimension reduction technique that maps high-dimensional data representation into a low-dimensional representation while retaining the pairwise distances between the data points as much as possible. More specifically, MDS is used to analyze similarities or proximities between pairs of data points.43 Function approximation algorithms A regression/prediction model tries to build the relationship between independent variables X (input) and dependent variables y (output).44 Moreover, it tries to find unknown parameters β such that the error (2) is minimized, given t predicted output yˆ as: ˆ yfX=(,). β (1) Let =− ˆ () i ii e yy be the difference between the dependent variable y and the predicted value yˆ. Therefore, the RMSE ξ over data samples of size n may be given as: ξ == 1 n. 2 1ei i n ∑ (2) International Journal of Nanomedicine 2015:10 submit your manuscript | www.dovepress.com Dovepress Dovepress 1122 Ojha et al Regression models such as linear regression (LReg), Gaussian process regression (GPReg), multilayer perceptron (MLP), and sequential minimal optimization regression (SMOReg) are as follows. lreg LReg is the simplest predictive model where independent variables ( |X|=n ×p), dependent variable ( |y|=p), with noise/error ( |ε|=p), may be written as: yx xxpx ii ipii T i =+ ++ =ε ββ ββ 12 12 . (3) gPreg The GPReg described by Rasmussen and Williams45 and Rasmussen and Nickisch46 is easily identified by its mean function m(x) and covariance function k(x, x′). This is a natural generalization of the Gaussian distribution, whose mean m and covariance k are a vector and a matrix, respectively. Gaussian distribution is defined over vectors, whereas the Gaussian process is defined over functions f. Therefore, we may write: fmk~().GP , (4) Considering a zero mean, linear and non-linear covariance functions may be given as: k(, )xx xx T ′′+,=α γ (5) kx xxx T (, )exp () ),(xx′′′=−−−      αγ 2 (6) where α and γ are the parameters of the basis function. MlP MLP is a feed-forward neural network having one or more hidden layers in between the input and output layers.47,48 A neuron in an MLP first computes a linear-weighted combination of real-valued inputs, and then limits its amplitude using a non-linear activation function. In the presented research, MLP was trained using the backpropagation algorithm49 and the resilient propagator.50 reduced error pruning tree Reduced error pruning (REP) tree is a fast decision tree learner. It builds a decision tree based on information gain or reduction of the variance and prunes it using reduced-error pruning with over-fitting.51,52 sMOreg Sequential minimal optimization (SMO), an algorithm for the training of support vector regression proposed by Smola and Schölkopf,53,54 and Schölkopf and Burges,55 is an extension of the SMO algorithm proposed by Platt56 for the support vector machine classifier. The idea of support vector regression is based on the computation of a linear regression function in a high-dimensional feature space where the input data are mapped using a non-linear function; support vector regression tries to minimize the generalization error in order to achieve generalized performance. The ensemble of function approximators Getting the best regression algorithm is not a trivial task. Apart from having a plethora of options as listed in the present section, one has to decide what the optimal sets of parameters for each algorithm are. There is generally very little guidance available to address the question of how to select an algorithm and adjust its parameters for a specific problem. In such cases, experimental tests can help the user to make decisions. Still, in many cases the obtained results are not satisfactory or even not acceptable. In such situations, the ensemble approach can be used. Basically, it relies on the assumption that the properly-modeled fusion of responses of several elementary predictors will produce more accurate results and reduce the regression error.57 Formally, let ∏ be a set of k predictors given as: Π= {} ff fk12 ,,,, (7) where fk indicates the state of the kth predictor. Each of the predictors is trained independently. The ensemble system fuses the outputs produced by the predictors in set ∏. In the simplest form, the ensemble can take the form of a simple average called the mean output regression, given as: Fx kfx i i k ′() ,()== ∑ 1 1 (8) where F ′ is an ensemble system. The natural advantage of this model is its simplicity, since the output of the ensemble can easily be obtained by simple mathematical transformation without the necessity of setting any additional parameters. On the other hand, the main drawback of this model is that it treats all the elementary predictors as equally important, regardless of their quality. Weak predictors affect the final output to the same degree as strong ones. As a result, the quality of the ensemble is close to the average of all its constituents. Better results can be obtained when the contribution of a particular predictor depends on its quality. The greater the accuracy of the predictor, the greater its weight International Journal of Nanomedicine 2015:10 submit your manuscript | www.dovepress.com Dovepress Dovepress 1123 Function approximation of Plga micro and nanoparticle dissolution rate in the ensemble. The ensemble method is therefore called the quality weighted output regression, given as: Fx kwf x ii i k ′() ,()== ∑ 1 1 (9) where wi i k=1 1= ∑ . In its simplest form, the weights should be counter-proportional to the RMSE of the given predictors. However, in more advanced algorithms, the weights can be set over the course of time, eg, be an application of evolutionary algorithms. Diversity of the ensemble There are several issues that have to be dealt with in order to make the application of the ensemble approach effective. One of the most essential issues is maintaining diversity among the predictors in the ensemble. Collecting a set of similar regression algorithms does not allow users to take any advantage from their fusion. Diversity can be ensured by applying one of the following procedures: 1. collecting predictors based on different models; 2. differentiating elementary predictor inputs. In the first approach, it is assumed that different regression algorithms naturally make errors that are uncorrelated, even when they are trained on the same data. The second group consists of algorithms that create an ensemble based on the same regression model, but diversity is caused by training each of them on data partitions (as diversity occurred due to data partition in the Bagging algorithm) or using heterogeneous feature sets (the techniques used in random subspace [RS] algorithms). rs algorithms RS is a method of constructing an ensemble of predictors where a pseudorandom procedure is used to select components of a feature vector separately for each ensemble constituent. The output of the ensemble is then obtained by averaging the outputs.58 Bagging algorithms Breiman59 introduced the bagging method, which is basically a combination of multiple predictors. At first, subsets are prepared by cutting the original dataset using bootstrapping. A sequence of predictors is then allowed to run over the subsets of the dataset. Finally, the results from each of the predictors are aggregated using voting in order to get the final results. This method is supposed to enhance the performance of ensemble systems and reduce variances in order to improve predictability.60,61 The evolutionary weighted ensemble The evolutionary weighted ensemble (EWE) is used to make decisions, based on Equation 8. The learning process searches for a set of weight that minimizes the RMSE of the ensemble, and for that purpose, the learning set is used. Therefore, the objective function for the learning procedure or the ensemble system can be written as: RMSEF′(),,,(),ww wNwf xy kjji i j k i N 12 2 1 =− ∑∑ (10) where xi, and yi denote the ith input–target pair in the learning set that consists of a total of N samples. We used the evolutionary algorithm,62 which processes a population of possible solutions encoded as chromosomes. An overview of the EWE training procedure is presented in Figure 1. The components of the EWE algorithms are defined as follows: Initial population The first step in the learning algorithm is generating an initial population. This consists of an arbitrarily chosen number of individuals with randomly selected weights that are scaled in order to ensure that their sum is 1. evaluation of the population Each individual is evaluated using an objective function. Obtained values determine the further behavior of the algorithm, especially selection procedures. selection of the elite The stability of the learning procedure is maintained by selecting two individuals with the smallest RMSE values. Those Input: learning set s – Population size g – The number of generations ∏ – a set of individual predictors repeat Initialize population for t = 1 to g do evaluate population over learning set select elite select parents Mutation crossover create offspring of the population end for until the stopping criterion satisfied. Figure 1 evolutionary weighted ensemble algorithm. International Journal of Nanomedicine 2015:10 submit your manuscript | www.dovepress.com Dovepress Dovepress 1124 Ojha et al individuals, called the elite, are not affected by mutation or crossover operators and join the offspring population. selection of the parents Only selected individuals participate in generating offspring for the new generation population. The selection is based on their fitness and is done in a probabilistic manner, ie, the smaller the RMSE of an individual, the greater the probability of its selection. Mutation The mutation operator of an evolutionary algorithm is supposed to ensure some amount of diversity within the population. In a classical implementation, it adds random noise to the chromosomes of selected individuals. crossover The crossover operator exchanges data between two selected parents and forms two new individuals and for that purpose, a standard 1-point crossover procedure can be used in which the cutting point is selected randomly. Offspring generation At the end of each generation the merging elite, the mutated individuals and children created by the crossover operator creates offspring. The new population substitutes the previous one and the entire process is repeated until a satisfactory solution is found, or the maximum iteration reached. Experiment setup and results To accomplish dimensionality reduction and identification of the corresponding regression model, the experiment was set up as follows: the dataset obtained for the PLGA dissolution profile had 300 features; therefore, the primary objective was to reduce the dimensions of the dataset. Hence, to accomplish this, the feature selection and feature extraction techniques discussed earlier were used. Subsequently, elementary prediction models were employed and their performances were assessed using ten-fold cross validation (10-CV) sets. Selection of the prediction model was based on the average of the RMSE computed over a set of ten results. In the final part of our experiment, we explored ensemble methods in order to exploit the elementary regression/prediction models. Feature selection method results After cleaning and preprocessing the dataset, a feature selection treatment was used in which we used a backward feature elimination technique with the GPReg, LReg, SMOReg and REP prediction models. The parameter settings of the prediction models are provided in Table 2. The combination of attributes that offers the lowest RMSE was considered as the optimal feature set. For example, the optimal feature set obtained using the GPReg, LReg, MLP, SMOReg and REP regression models are 18, 32, 31, 30, and 31 with RMSE values of (resulting from a normalized dataset) 0.143, 0.156, 0.121, 0.153, and 0.126, respectively. The backward feature elimination results were convening in terms of RMSE. Therefore, for each of the predictors, we selected the feature sets with the smaller attributes, ie, set with ten, five, and one attribute. We have stochastic feature selection techniques such as correlation-based, classifier-based, and wrapper-based methods. These feature selection methods were used to determine the merits (predictability) of different combinations of features. After assigning the merits of the several sets of features, the best first search (BFS) and the greedy search (greedy) methods were used to select the desired optimal feature set. Interestingly, in the present problem, when we used correlation-based feature selection, both the BFS and greedy searches produced identical feature sets with five attributes. The classifier-based feature selection was patched with GPReg, MLP, and LR eg, respectively, in order to evaluate the merits of the feature set. Subsequently, BFS and greedy searches were used to determine the optimal feature set. Therefore, we had class-GPReg-BFS, class-GPReg-greedy, class-MLP-BFS, class-MLP-greedy, class-LReg-BFS, and class-LReg-greedy feature selection methods, indicating a classifier-based method with GPReg as a feature set merit evaluator and BFS as the method to select the optimal feature set. Similarly, wrapper-GP-greedy, wrapper-MLPgreedy, and wrapper-LReg-greedy indicate a combination of wrapper-based feature selections, where GPReg, MLP, and LReg were used to evaluate the feature set. Interestingly, both BFS and greedy searches offered identical feature sets. Table 2 Parameters setting of the respective regression models used for the feature selection and feature extraction experiments Predictor Parameters gPreg rBF kernel, gamma value = 1.0 lreg – MlP Three-layer MlP, hidden layer nodes - 50, learning rate - 0.3, momentum rate - 0.2 sMOreg Polynomial kernel, epsilon value - 0.001, tolerance level - 0.001 reP tree Max depth – no restriction Abbreviations: gPreg, gaussian process regression; rBF, radial basis function; lreg, linear regression; MlP, multilayer perception; sMOreg, sequential minimal optimization regression; reP, reduced error pruning; –, no such parameter. International Journal of Nanomedicine 2015:10 submit your manuscript | www.dovepress.com Dovepress Dovepress 1125 Function approximation of Plga micro and nanoparticle dissolution rate A list of feature selection methods and the corresponding selected features are illustrated in Table 3. results of the feature extraction technique Unlike feature selection, feature extraction finds a new set of reduced features by computing linear or non-linear combinations of features from the available dataset. A comprehensive result is presented in Table 4, which illustrates the performance of feature extraction methods and regression models. Dimensionality reduction tools offered by van der Maaten et al20 were used for the feature extraction. PCA and FA linear dimensionality reduction methods, and non-linear dimensionality reduction methods such as kPCA and MDS were used to reduce the dimensions of the dataset from 300 to 50, 30, 20, 10, and 5. ICA was used to reduce the dimension of the dataset from 300 to 50. Results obtained using ICA are as follows. The mean RMSE and variance corresponding to GPReg, LReg, MLP, and SMOReg are 14.83, 17.23, 13.94, and 17.92 and 3.61, 2.34, 2.77, and 2.87, respectively. It may be observed from Table 4 that lower dimensions offer less significant improvement in terms of RMSE. However, if we compare the best results (the result of reducing the dimension to 50) of PCA (an RMSE of 13.59 corresponding to MLP) and ICA (an RMSE of 13.94 corresponding to MLP) with the result using all features (an RMSE of 16.812 corresponding to GPReg), it is evident that reducing the dimension significantly improves the performance of the prediction model. Examining Figure 2, an RMSE and variance comparison between chosen regression models applied on the dataset reduced it to a dimension of 50 using ICA, PCA, FA, kPCA, and MDS feature extraction techniques; we may conclude that the feature extraction using PCA performed best, both in terms of RMSE and variance, when the MLP regression model was used, whereas the feature extraction using ICA was second to PCA when MLP was used. When it came to GPReg, ICA had an edge over PCA. The regression model and ensemble results In order to identify a suitable regression model, we chose several regression models. The parameter settings corresponding to the regression models are given in Table 3. A comprehensive feature selection result using 10-CV is presented in Table 3. Examining Table 3, we may therefore draw the following conclusions. First of all, in Table 3, we arranged the feature selection methods according to ascending order of the number of features selected by the feature selection methods; the first row of Table 3 that indicates no feature selection (ie, all 300 features were used), is exceptional. We compared the results of the prediction models arranged in the columns in Table 3. The feature selection process was able to find the most significant features that influenced the drug release rate. It may be observed that feature vectors from all the mentioned feature selection methods obtained a reduced Table 3 experimental results for 10-cV datasets prepared with distinct random partitions of the complete dataset using feature selection technique (Identification of regression model) Selection method Selected features GPReg LReg MLP REP SMOReg No selection 300 16.81 17.07 18.57 13.05 17.95 BFe 1 27.47 26.61 28.33 24.37 26.97 BFe 5 17.11 23.45 23.11 14.23 23.38 cFs 5 20.80 25.08 22.41 18.31 25.42 class-MlP-greedy 7 17.96 25.03 22.26 14.96 25.35 BFe 10 15.93 19.98 21.00 13.19 19.53 class-MlP-BFs 15 15.88 22.90 16.83 13.91 24.23 Wrapper-gPreg-greedy 15 14.88 20.22 15.20 13.34 20.86 class-gPreg-BFs 16 18.46 23.07 19.71 14.19 23.69 class-gPreg-greedy 19 15.06 19.05 15.61 14.03 19.68 Wrapper-MlP-greedy 19 16.44 24.01 20.42 14.26 24.85 Wrapper-lreg-greedy 24 15.91 17.46 17.03 13.54 18.02 BFe Optimal* 15.71 17.85 17.82 13.90 17.88 class-lreg-BFs 31 15.95 16.92 15.63 14.00 17.58 class-lreg-greedy 37 16.31 17.14 16.27 14.02 17.69 Notes: Values are the average of ten rMse. *Optimal set of attributes for the gPreg, lreg, MlP, reP and sMOreg regression models are 18, 32, 31, 31, and 30, respectively. Abbreviations: 10-cV, ten-fold cross-validation; gPreg, gaussian process regression; lreg, linear regression; MlP, multilayer perception; reP, reduced error pruning; SMOReg, sequential minimal optimization; No, number; BFE, backward feature elimination; CFS, correlation-based feature selection; BFS, best fit search; wrapper, wrapper feature selection; greedy, greedy search; class, classifier-based feature selection. International Journal of Nanomedicine 2015:10 submit your manuscript | www.dovepress.com Dovepress Dovepress 1126 Ojha et al set of the most influential features. Therefore, a general theory may be drawn about how and which features are the most dominant with regard to the PLGA drug release rate. It is worth mentioning that the best result presented by Szlęk et al21 is an RMSE of 15.4, considering eleven selected features using MLP and 17 features with an RMSE of 14.3 using MLP. From Table 3, it may be observed that when considering all 300 features, the best result we can achieve is by using REP, resulting in an RMSE of 13.05 (the average of the 10-CV result). Therefore, any regression Table 4 experimental results for 10-cV datasets prepared with distinct random partitions of the complete dataset using feature extraction techniques Feature extraction method Regression model Dimension reduction 5 10 20 30 50 Mean VAR Mean VAR Mean VAR Mean VAR Mean VAR linear method Pca gPreg 28.88 1.62 27.22 3.00 24.80 3.85 19.82 2.49 16.08 3.16 lreg 29.55 1.74 29.22 1.70 27.73 2.21 23.93 1.63 17.17 2.79 MlP 30.36 3.36 29.77 6.37 26.58 3.98 19.89 2.27 13.59 1.56 sMOreg 30.14 3.17 29.78 3.62 27.95 2.67 24.31 1.89 17.66 3.09 Fa gPreg 29.23 1.77 28.56 2.67 28.31 3.34 28.30 3.42 28.26 3.31 lreg 29.97 1.77 29.97 1.77 29.97 1.77 29.97 1.77 29.98 1.82 MlP 30.64 2.02 30.50 1.91 31.01 1.83 30.93 2.30 30.91 0.77 sMOreg 30.28 3.45 30.28 3.45 30.26 3.37 30.29 3.44 30.28 3.46 Non-linear method Kernel Pca gPreg 28.60 1.68 27.08 2.12 24.96 1.96 24.32 2.17 22.81 4.43 lreg 29.31 1.52 28.05 1.78 25.35 2.05 25.17 2.23 22.98 4.27 MlP 29.81 3.57 29.65 7.94 27.07 4.09 25.97 5.52 25.27 8.49 sMOreg 29.43 1.41 28.68 1.65 25.90 1.70 25.79 2.00 23.24 4.76 MDs gPreg 28.91 2.17 28.73 2.47 28.41 3.16 28.24 3.17 28.16 3.27 lreg 29.56 1.86 29.21 2.08 29.19 2.08 29.11 1.92 29.14 2.04 MlP 30.42 3.71 29.38 4.11 29.93 3.10 30.01 4.53 29.98 4.42 sMOreg 29.98 2.62 29.64 2.55 29.64 2.76 29.66 2.85 29.65 2.89 Note: Mean and variance (Var) is computed on ten rMse obtained. Abbreviations: 10-cV, ten-fold cross-validation; rMse, root mean square error; Pca, principal component analysis; Fa, factor analysis; MDs, multidimensional scaling; gPreg, gaussian process regression; lreg, linear regression; MlP, multilayer perception; sMOreg, sequential minimal optimization regression. model tested with a reduced feature set must compete with this result. In our study, the best result was obtained with the feature set using the wrapper-GPReg-greedy method with RSME of 14.88, 20.22, 15.20, 13.31, and 20.86 using the GPReg, LReg, MLP, REP and SMOReg elementary models, respectively. Therefore, we may consider the features “fused ring count”, “heteroaromatic ring count”, “largest ring system size”, “chain atom count”, “chain bond count”, and “quaternary structure” from the protein descriptors group of features; “PVA concentration inner phase”, “PVA Figure 2 results of the feature extraction experiment for the reduced dimension set of 30 features: a comparison between the regression models. a comparison using average rMse (A); a comparison using variances (B). Abbreviations: rMse, root mean square error; Ica, independent component analysis; Pca, principle component analysis; Fa, factor analysis; kPca, kernel Pca; MDs, multidimensional scaling; gPreg, gaussian process regression; lreg, linear regression; MlP, multilayer perception; sMOreg, sequential minimal optimization regression.  ,&$ 3&$ )$ N3&$ 0'6 ,&$ 3&$ )$ N3&$ 0'6        $YHUDJH506( $           9DULDQFH % *35HJ /5HJ 0/3 6025HJ International Journal of Nanomedicine 2015:10 submit your manuscript | www.dovepress.com Dovepress Dovepress 1127 Function approximation of Plga micro and nanoparticle dissolution rate concentration outer phase”, “PVA molecular weight”, and “PLGA to placticizer” from the formulation characteristics group of features; and “acetylsalicylic acid”, “Szeged index”, and “pH=12 logD” from the plasticizer group of features. From the emulsifier group, we have “a(yy)” and “time in days” as being the most influential feature sets obtained using wrapper-GPReg-greedy experiment. A complete list of the feature names can be found in Szlęk et al.2 After we obtained the best features, we resorted to using ensemble techniques. A comprehensive comparison of the results obtained using the ensemble methods and other elementary regression models is given in Table 5. From the results presented in Table 5, it is evident that some of the listed ensemble methods provides better results than that of the result produced by the best elementary predictor ie, reduced error pruning tree. The average RMSE obtained by ensemble such as RS using REP, RS using MLP, RS using GPReg, Bagging using REP, Bagging using MLP, mean output regression, quality weighted output regression, and EWE are 13.85, 18.20, 18.72, 11.49, 12.30, 10.43, 10.06, and 7.67, respectively. Discussion and analysis In this article, experimental results obtained using both feature selection and feature extraction techniques are offered. The primary objective of the experiments was to find the lowest RMSE. In addition, we took advantage of the feature selection methods to obtain the best set of features. Our benchmark for the present experiment was the RMSE obtained using the complete set of features, ie, 300 features, and the results obtained by Szlęk et al.21 The results obtained by the feature selection, feature extraction, and ensemble experiments are provided in Tables 3–5, respectively. The wrapper-based feature selection technique provided us the set of the most significant features. On the other hand, PCA offered a new set of features with solutions that were better than the solutions obtained with the complete dataset. The ensemble methods were only used for the feature selection methods. The ensemble methods enabled us to exploit all the evaluated regression models. Therefore, the best result (lowest RMSE) out of all the trained regressors was obtained using the EWE ensemble method. As mentioned above, predicting the PLGA dissolution rate is an important problem for the pharmaceutical industry. More significantly, identifying the influencing factors (features) is crucial for predicting the PLGA dissolution rate. Conclusion Analyzing the effectiveness of the ensemble methods should be based on a comparison of the results obtained using the best elementary predictors. In our case, among the tested simple predictors, the lowest RMSE was reached with REP (13.34). The ensemble methods should improve regression accuracy over the best elementary predictor. The EWE ensemble method offered the lowest RMSE, which proves that in certain cases, combining the outputs of several predictors allows us to improve overall accuracy. It is essential to ensure diversity among the ensemble’s constituents. Among the tested techniques, an ensemble of five heterogeneous regression algorithms provided the best results. Weighting their outputs was the most effective when weights were set Table 5 a comprehensive conclusion of the results obtained from each regression model, including the ensemble techniques used Ensemble method RMSE Graphical representation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 eWe 7.67 QWOr 10.06 MOr 10.43 Bagging-reP 11.49 Bagging-MlP 12.30 rs-reP 13.85 rs-MlP 18.20 rs-gPreg 18.72 sMOreg 20.86 reP tree 13.34 MlP 15.20 lreg 20.22 gPreg 14.88 Note: We have selected the feature set that was obtained using wrapper-gPreg-greedy search. Abbreviations: rMse, root mean square error; eWe, evolutionary weighted ensemble; QWOr, quality weighted output regression; MOr, mean output regression; bagging, ; MlP, multilayer perception; rs, random subspace; reP, reduced error pruning; sMOreg, sequential minimal optimization regression; lreg, linear regression; gPreg, gaussian process regression.