CNN Architecture for Posture Classification on Small Data

IFAC PapersOnLine 58-9 (2024) 299–304 ScienceDirect ScienceDirect Available online at www.sciencedirect.com 2405-8963 Copyright © 2024 The Authors. This is an open access article under the CC BY-NC-ND license . Peer review under responsibility of International Federation of Automatic Control. 10.1016/j.ifacol.2024.07.413 10.1016/j.ifacol.2024.07.413 2405-8963 CNN Architecture for Posture Classification on Small Data Michaela Mes´aroˇsov´a ∗Ondrej Mih´alik ∗Miroslav Jirgl ∗ ∗Department of Control and Instrumentation, Faculty of Electrical Engineering, Brno University of Technology, Brno, Czech republic (e-mail: [email protected]). Abstract: A convolutional neural network is often mentioned as one of the deep learning methods that requires a large amount of training data. Questioning this belief, this paper explores the applicability of classification based on a shallow net structure trained on a small data set in the context of patient posture classification based on data from a pressure mattress. Designing a CNN often presents a complex problem, especially without a universally applicable approach, allowing many diverse structural possibilities and training settings. We tested various training options and layer configurations to provide an overview of influential parameters for posture classification. Experiments show encouraging results with the leave-one-out crossvalidation accuracy of 93.1% of one of the evaluated CNN structures and its hyperparameter settings. Keywords: CNN, fine tuning, network structure, optimization, posture classification 1. INTRODUCTION Deep learning methods benefit greatly from very large data sets. However, in some situations, it might be difficult to gather enough data, if the data-acquisition process is time-consuming or laborious. Choosing the structure and parameters of a CNN that represents images and their stored information uniquely and correctly is one of the most important aspects of achieving a high CNN performance. Typically the CNN configuration parameters are divided into two categories: those concerning the architecture and those concerning the training process, see Ferreyra-Ramirez et al. (2019). The specific initialization of the parameters often has a significant impact on how long it takes for the training process to find a solution and on the generalization ability of the resulting trained network. Regretfully, as stated in Bishop and Bishop (2023), there is not much theory to help determine an initialization strategy. For this reason, we offer a compact overview of a selection of settings and modifiable parameters and their impact on the classification results. 2. NETWORK TRAINING OPTIONS The right design of the network is crucial for the application where the CNN is to be applied. However the same importance lies in the task of discovering appropriate training parameters that lead to the best-performing classifier. We introduce some of these parameters along with an overview of their role in the training process. 2.1 Optimization algorithms In MATLAB 2023, there are available the following four optimization algorithms: • SGDM, •RMSProp, •Adam, •L-BGFS Although there exist many more algorithms, they are often derivatives of those listed above, hence we will leave it for the reader to evaluate their performance after the most suitable base approach was selected. Only the first three will be taken into consideration in this paper. The Stochastic Gradient Descent with Momentum or SGDM is an extension of the Stochastic gradient descent (SGD) algorithm that updates the network parameters to minimize the loss function by taking small steps at each iteration in the direction of the negative gradient of the loss, θℓ+1 =θℓ−α∇E(θℓ)+γ(θℓ−θℓ−1),(1) where αis the learning rate,γis the momentum value, ℓis the iteration number, θis the parameter vector, and E(θ) is the loss function. The gradient is evaluated using a subset of the training data, also called a mini-batch, where a different subset is used at each iteration, for detailed explanation see Bishop and Bishop (2023) and Beale et al. (2023). The entire pass over the training data is called an epoch. The contribution of the previous gradient step to the current iteration is determined by the learning rate αand the momentum term γ, which in addition reduces the oscillation around the optimum that might occur in the case of a very steep negative gradient. The unpublished Root Mean Square Propagation or RMSProp is designed to accelerate the optimization process, and thus, decrease the number of function evaluations to reach the optimum. This is achieved by using divergent learning rates for each weight that are able to adapt automatically to the optimization of the loss function. The learning rate for a particular weight is divided by a CNN Architecture for Posture Classification on Small Data Michaela Mes´aroˇsov´a ∗Ondrej Mih´alik ∗Miroslav Jirgl ∗ ∗Department of Control and Instrumentation, Faculty of Electrical Engineering, Brno University of Technology, Brno, Czech republic (e-mail: [email protected]). Abstract: A convolutional neural network is often mentioned as one of the deep learning methods that requires a large amount of training data. Questioning this belief, this paper explores the applicability of classification based on a shallow net structure trained on a small data set in the context of patient posture classification based on data from a pressure mattress. Designing a CNN often presents a complex problem, especially without a universally applicable approach, allowing many diverse structural possibilities and training settings. We tested various training options and layer configurations to provide an overview of influential parameters for posture classification. Experiments show encouraging results with the leave-one-out crossvalidation accuracy of 93.1% of one of the evaluated CNN structures and its hyperparameter settings. Keywords: CNN, fine tuning, network structure, optimization, posture classification 1. INTRODUCTION Deep learning methods benefit greatly from very large data sets. However, in some situations, it might be difficult to gather enough data, if the data-acquisition process is time-consuming or laborious. Choosing the structure and parameters of a CNN that represents images and their stored information uniquely and correctly is one of the most important aspects of achieving a high CNN performance. Typically the CNN configuration parameters are divided into two categories: those concerning the architecture and those concerning the training process, see Ferreyra-Ramirez et al. (2019). The specific initialization of the parameters often has a significant impact on how long it takes for the training process to find a solution and on the generalization ability of the resulting trained network. Regretfully, as stated in Bishop and Bishop (2023), there is not much theory to help determine an initialization strategy. For this reason, we offer a compact overview of a selection of settings and modifiable parameters and their impact on the classification results. 2. NETWORK TRAINING OPTIONS The right design of the network is crucial for the application where the CNN is to be applied. However the same importance lies in the task of discovering appropriate training parameters that lead to the best-performing classifier. We introduce some of these parameters along with an overview of their role in the training process. 2.1 Optimization algorithms In MATLAB 2023, there are available the following four optimization algorithms: •SGDM, •RMSProp, •Adam, •L-BGFS Although there exist many more algorithms, they are often derivatives of those listed above, hence we will leave it for the reader to evaluate their performance after the most suitable base approach was selected. Only the first three will be taken into consideration in this paper. The Stochastic Gradient Descent with Momentum or SGDM is an extension of the Stochastic gradient descent (SGD) algorithm that updates the network parameters to minimize the loss function by taking small steps at each iteration in the direction of the negative gradient of the loss, θℓ+1 =θℓ−α∇E(θℓ)+γ(θℓ−θℓ−1),(1) where αis the learning rate,γis the momentum value, ℓis the iteration number, θis the parameter vector, and E(θ) is the loss function. The gradient is evaluated using a subset of the training data, also called a mini-batch, where a different subset is used at each iteration, for detailed explanation see Bishop and Bishop (2023) and Beale et al. (2023). The entire pass over the training data is called an epoch. The contribution of the previous gradient step to the current iteration is determined by the learning rate αand the momentum term γ, which in addition reduces the oscillation around the optimum that might occur in the case of a very steep negative gradient. The unpublished Root Mean Square Propagation or RMSProp is designed to accelerate the optimization process, and thus, decrease the number of function evaluations to reach the optimum. This is achieved by using divergent learning rates for each weight that are able to adapt automatically to the optimization of the loss function. The learning rate for a particular weight is divided by a CNN Architecture for Posture Classification on Small Data Michaela Mes´aroˇsov´a ∗Ondrej Mih´alik ∗Miroslav Jirgl ∗ ∗Department of Control and Instrumentation, Faculty of Electrical Engineering, Brno University of Technology, Brno, Czech republic (e-mail: [email protected]). Abstract: A convolutional neural network is often mentioned as one of the deep learning methods that requires a large amount of training data. Questioning this belief, this paper explores the applicability of classification based on a shallow net structure trained on a small data set in the context of patient posture classification based on data from a pressure mattress. Designing a CNN often presents a complex problem, especially without a universally applicable approach, allowing many diverse structural possibilities and training settings. We tested various training options and layer configurations to provide an overview of influential parameters for posture classification. Experiments show encouraging results with the leave-one-out crossvalidation accuracy of 93.1% of one of the evaluated CNN structures and its hyperparameter settings. Keywords: CNN, fine tuning, network structure, optimization, posture classification 1. INTRODUCTION Deep learning methods benefit greatly from very large data sets. However, in some situations, it might be difficult to gather enough data, if the data-acquisition process is time-consuming or laborious. Choosing the structure and parameters of a CNN that represents images and their stored information uniquely and correctly is one of the most important aspects of achieving a high CNN performance. Typically the CNN configuration parameters are divided into two categories: those concerning the architecture and those concerning the training process, see Ferreyra-Ramirez et al. (2019). The specific initialization of the parameters often has a significant impact on how long it takes for the training process to find a solution and on the generalization ability of the resulting trained network. Regretfully, as stated in Bishop and Bishop (2023), there is not much theory to help determine an initialization strategy. For this reason, we offer a compact overview of a selection of settings and modifiable parameters and their impact on the classification results. 2. NETWORK TRAINING OPTIONS The right design of the network is crucial for the application where the CNN is to be applied. However the same importance lies in the task of discovering appropriate training parameters that lead to the best-performing classifier. We introduce some of these parameters along with an overview of their role in the training process. 2.1 Optimization algorithms In MATLAB 2023, there are available the following four optimization algorithms: •SGDM, •RMSProp, •Adam, •L-BGFS Although there exist many more algorithms, they are often derivatives of those listed above, hence we will leave it for the reader to evaluate their performance after the most suitable base approach was selected. Only the first three will be taken into consideration in this paper. The Stochastic Gradient Descent with Momentum or SGDM is an extension of the Stochastic gradient descent (SGD) algorithm that updates the network parameters to minimize the loss function by taking small steps at each iteration in the direction of the negative gradient of the loss, θℓ+1 =θℓ−α∇E(θℓ)+γ(θℓ−θℓ−1),(1) where αis the learning rate,γis the momentum value, ℓis the iteration number, θis the parameter vector, and E(θ) is the loss function. The gradient is evaluated using a subset of the training data, also called a mini-batch, where a different subset is used at each iteration, for detailed explanation see Bishop and Bishop (2023) and Beale et al. (2023). The entire pass over the training data is called an epoch. The contribution of the previous gradient step to the current iteration is determined by the learning rate αand the momentum term γ, which in addition reduces the oscillation around the optimum that might occur in the case of a very steep negative gradient. The unpublished Root Mean Square Propagation or RMSProp is designed to accelerate the optimization process, and thus, decrease the number of function evaluations to reach the optimum. This is achieved by using divergent learning rates for each weight that are able to adapt automatically to the optimization of the loss function. The learning rate for a particular weight is divided by a CNN Architecture for Posture Classification on Small Data Michaela Mes´aroˇsov´a ∗Ondrej Mih´alik ∗Miroslav Jirgl ∗ ∗ Department of Control and Instrumentation, Faculty of Electrical Engineering, Brno University of Technology, Brno, Czech republic (e-mail: Michaela.Mesaros[email protected]). Abstract: A convolutional neural network is often mentioned as one of the deep learning methods that requires a large amount of training data. Questioning this belief, this paper explores the applicability of classification based on a shallow net structure trained on a small data set in the context of patient posture classification based on data from a pressure mattress. Designing a CNN often presents a complex problem, especially without a universally applicable approach, allowing many diverse structural possibilities and training settings. We tested various training options and layer configurations to provide an overview of influential parameters for posture classification. Experiments show encouraging results with the leave-one-out crossvalidation accuracy of 93.1% of one of the evaluated CNN structures and its hyperparameter settings. Keywords: CNN, fine tuning, network structure, optimization, posture classification 1. INTRODUCTION Deep learning methods benefit greatly from very large data sets. However, in some situations, it might be difficult to gather enough data, if the data-acquisition process is time-consuming or laborious. Choosing the structure and parameters of a CNN that represents images and their stored information uniquely and correctly is one of the most important aspects of achieving a high CNN performance. Typically the CNN configuration parameters are divided into two categories: those concerning the architecture and those concerning the training process, see Ferreyra-Ramirez et al. (2019). The specific initialization of the parameters often has a significant impact on how long it takes for the training process to find a solution and on the generalization ability of the resulting trained network. Regretfully, as stated in Bishop and Bishop (2023), there is not much theory to help determine an initialization strategy. For this reason, we offer a compact overview of a selection of settings and modifiable parameters and their impact on the classification results. 2. NETWORK TRAINING OPTIONS The right design of the network is crucial for the application where the CNN is to be applied. However the same importance lies in the task of discovering appropriate training parameters that lead to the best-performing classifier. We introduce some of these parameters along with an overview of their role in the training process. 2.1 Optimization algorithms In MATLAB 2023, there are available the following four optimization algorithms: •SGDM, •RMSProp, •Adam, •L-BGFS Although there exist many more algorithms, they are often derivatives of those listed above, hence we will leave it for the reader to evaluate their performance after the most suitable base approach was selected. Only the first three will be taken into consideration in this paper. The Stochastic Gradient Descent with Momentum or SGDM is an extension of the Stochastic gradient descent (SGD) algorithm that updates the network parameters to minimize the loss function by taking small steps at each iteration in the direction of the negative gradient of the loss, θℓ+1 =θℓ−α∇E(θℓ)+γ(θℓ−θℓ−1),(1) where αis the learning rate,γis the momentum value, ℓis the iteration number, θis the parameter vector, and E(θ) is the loss function. The gradient is evaluated using a subset of the training data, also called a mini-batch, where a different subset is used at each iteration, for detailed explanation see Bishop and Bishop (2023) and Beale et al. (2023). The entire pass over the training data is called an epoch. The contribution of the previous gradient step to the current iteration is determined by the learning rate αand the momentum term γ, which in addition reduces the oscillation around the optimum that might occur in the case of a very steep negative gradient. The unpublished Root Mean Square Propagation or RMSProp is designed to accelerate the optimization process, and thus, decrease the number of function evaluations to reach the optimum. This is achieved by using divergent learning rates for each weight that are able to adapt automatically to the optimization of the loss function. The learning rate for a particular weight is divided by a CNN Architecture for Posture Classification on Small Data Michaela Mes´aroˇsov´a ∗ Ondrej Mih´alik ∗ Miroslav Jirgl ∗ ∗Department of Control and Instrumentation, Faculty of Electrical Engineering, Brno University of Technology, Brno, Czech republic (e-mail: [email protected]). Abstract: A convolutional neural network is often mentioned as one of the deep learning methods that requires a large amount of training data. Questioning this belief, this paper explores the applicability of classification based on a shallow net structure trained on a small data set in the context of patient posture classification based on data from a pressure mattress. Designing a CNN often presents a complex problem, especially without a universally applicable approach, allowing many diverse structural possibilities and training settings. We tested various training options and layer configurations to provide an overview of influential parameters for posture classification. Experiments show encouraging results with the leave-one-out crossvalidation accuracy of 93.1% of one of the evaluated CNN structures and its hyperparameter settings. Keywords: CNN, fine tuning, network structure, optimization, posture classification 1. INTRODUCTION Deep learning methods benefit greatly from very large data sets. However, in some situations, it might be difficult to gather enough data, if the data-acquisition process is time-consuming or laborious. Choosing the structure and parameters of a CNN that represents images and their stored information uniquely and correctly is one of the most important aspects of achieving a high CNN performance. Typically the CNN configuration parameters are divided into two categories: those concerning the architecture and those concerning the training process, see Ferreyra-Ramirez et al. (2019). The specific initialization of the parameters often has a significant impact on how long it takes for the training process to find a solution and on the generalization ability of the resulting trained network. Regretfully, as stated in Bishop and Bishop (2023), there is not much theory to help determine an initialization strategy. For this reason, we offer a compact overview of a selection of settings and modifiable parameters and their impact on the classification results. 2. NETWORK TRAINING OPTIONS The right design of the network is crucial for the application where the CNN is to be applied. However the same importance lies in the task of discovering appropriate training parameters that lead to the best-performing classifier. We introduce some of these parameters along with an overview of their role in the training process. 2.1 Optimization algorithms In MATLAB 2023, there are available the following four optimization algorithms: •SGDM, •RMSProp, •Adam, •L-BGFS Although there exist many more algorithms, they are often derivatives of those listed above, hence we will leave it for the reader to evaluate their performance after the most suitable base approach was selected. Only the first three will be taken into consideration in this paper. The Stochastic Gradient Descent with Momentum or SGDM is an extension of the Stochastic gradient descent (SGD) algorithm that updates the network parameters to minimize the loss function by taking small steps at each iteration in the direction of the negative gradient of the loss, θℓ+1 =θℓ−α∇E(θℓ)+γ(θℓ−θℓ−1),(1) where αis the learning rate,γis the momentum value, ℓis the iteration number, θis the parameter vector, and E(θ) is the loss function. The gradient is evaluated using a subset of the training data, also called a mini-batch, where a different subset is used at each iteration, for detailed explanation see Bishop and Bishop (2023) and Beale et al. (2023). The entire pass over the training data is called an epoch. The contribution of the previous gradient step to the current iteration is determined by the learning rate αand the momentum term γ, which in addition reduces the oscillation around the optimum that might occur in the case of a very steep negative gradient. The unpublished Root Mean Square Propagation or RMSProp is designed to accelerate the optimization process, and thus, decrease the number of function evaluations to reach the optimum. This is achieved by using divergent learning rates for each weight that are able to adapt automatically to the optimization of the loss function. The learning rate for a particular weight is divided by a CNN Architecture for Posture Classification on Small Data Michaela Mes´aroˇsov´a ∗Ondrej Mih´alik ∗Miroslav Jirgl ∗ ∗Department of Control and Instrumentation, Faculty of Electrical Engineering, Brno University of Technology, Brno, Czech republic (e-mail: [email protected]). Abstract: A convolutional neural network is often mentioned as one of the deep learning methods that requires a large amount of training data. Questioning this belief, this paper explores the applicability of classification based on a shallow net structure trained on a small data set in the context of patient posture classification based on data from a pressure mattress. Designing a CNN often presents a complex problem, especially without a universally applicable approach, allowing many diverse structural possibilities and training settings. We tested various training options and layer configurations to provide an overview of influential parameters for posture classification. Experiments show encouraging results with the leave-one-out crossvalidation accuracy of 93.1% of one of the evaluated CNN structures and its hyperparameter settings. Keywords: CNN, fine tuning, network structure, optimization, posture classification 1. INTRODUCTION Deep learning methods benefit greatly from very large data sets. However, in some situations, it might be difficult to gather enough data, if the data-acquisition process is time-consuming or laborious. Choosing the structure and parameters of a CNN that represents images and their stored information uniquely and correctly is one of the most important aspects of achieving a high CNN performance. Typically the CNN configuration parameters are divided into two categories: those concerning the architecture and those concerning the training process, see Ferreyra-Ramirez et al. (2019). The specific initialization of the parameters often has a significant impact on how long it takes for the training process to find a solution and on the generalization ability of the resulting trained network. Regretfully, as stated in Bishop and Bishop (2023), there is not much theory to help determine an initialization strategy. For this reason, we offer a compact overview of a selection of settings and modifiable parameters and their impact on the classification results. 2. NETWORK TRAINING OPTIONS The right design of the network is crucial for the application where the CNN is to be applied. However the same importance lies in the task of discovering appropriate training parameters that lead to the best-performing classifier. We introduce some of these parameters along with an overview of their role in the training process. 2.1 Optimization algorithms In MATLAB 2023, there are available the following four optimization algorithms: •SGDM, •RMSProp, •Adam, •L-BGFS Although there exist many more algorithms, they are often derivatives of those listed above, hence we will leave it for the reader to evaluate their performance after the most suitable base approach was selected. Only the first three will be taken into consideration in this paper. The Stochastic Gradient Descent with Momentum or SGDM is an extension of the Stochastic gradient descent (SGD) algorithm that updates the network parameters to minimize the loss function by taking small steps at each iteration in the direction of the negative gradient of the loss, θℓ+1 =θℓ−α∇E(θℓ)+γ(θℓ−θℓ−1),(1) where αis the learning rate,γis the momentum value, ℓis the iteration number, θis the parameter vector, and E(θ) is the loss function. The gradient is evaluated using a subset of the training data, also called a mini-batch, where a different subset is used at each iteration, for detailed explanation see Bishop and Bishop (2023) and Beale et al. (2023). The entire pass over the training data is called an epoch. The contribution of the previous gradient step to the current iteration is determined by the learning rate αand the momentum term γ, which in addition reduces the oscillation around the optimum that might occur in the case of a very steep negative gradient. The unpublished Root Mean Square Propagation or RMSProp is designed to accelerate the optimization process, and thus, decrease the number of function evaluations to reach the optimum. This is achieved by using divergent learning rates for each weight that are able to adapt automatically to the optimization of the loss function. The learning rate for a particular weight is divided by a Copyright © 2024 The Authors. This is an open access article under the CC BY-NC-ND license ( https://creativecommons.org/licenses/by-nc-nd/4.0/ ) 300 Michaela Mesárošová et al. / IFAC PapersOnLine 58-9 (2024) 299–304 running average of the magnitudes of recent gradients for the corresponding weight. vℓ=β2vℓ−1+ (1 −β2)[∇E(θℓ)]2(2) θℓ+1 =θℓ−α∇E(θℓ) √vℓ+ϵ,(3) where vℓis the moving average, β2is the squared gradient decay factor of the moving average, and ϵis a small constant to avoid division by zero, for more detail see Tieleman and Hinton (2012). As stated by Kingma and Ba (2015), RMSProp is suited for online and non-stationary settings. Adaptive moment estimation, also called Adam, was first introduced in Kingma and Ba (2015). It is an efficient stochastic optimization that requires first-order gradients. Individual adaptive learning rates are again maintained for all parameters separately. Updates are directly estimated by using a running average of the first and the second moments of the gradients. Moving averages are calculated as follows: mℓ=β1mℓ−1+ (1 −β1)∇E(θℓ) (4) vℓ=β2vℓ−1+ (1 −β2)[∇E(θℓ)]2(5) mℓ=mℓ/(1 −βℓ 1) (6) vℓ=vℓ/(1 −βℓ 2) (7) where β1and β2are the exponential decay rates for the moment estimates. The network parameters then are updated as θℓ+1 =θℓ−αmℓ √vℓ+ϵ(8) If gradients are similar throughout many iterations updates of the weights are able to gain momentum in a certain direction by using a moving average of the gradient. The Adam algorithm is suitable for working with large amounts of data for its small memory requirements. 2.2 Learning rate Learning rate, also referred to as step size, determines the rate of change of the weights. The initial learning rate has a significant impact on the learning speed and the overall training development. When the learning rate is too large, the training error may inadvertently increase rather than decrease, yet with a value too small, training is substantially slower and may become stuck with a high training error, for more detail see Goodfellow et al. (2016). Further remarks from Bengio (2012) and Reed and Marks (1999) identify the learning rate as the most important hyperparameter. Its typical default value is set to 0.01. However, additional tuning is always recommended. 2.3 Regularization A reasonable approach to problem-solving would be to adapt the model complexity according to the complexity of the problem. One way to control generalization and avoid the overfitting phenomenon is by regularization. To include regularization in model training, a penalty function is added to the loss function E(θ), discouraging the parameters from having large magnitudes.  E(θ)=E(θ)+λΩ (9) where Ω is the regularization function and λis the multiplicative parameter. An optimal network is one in which a compromise between the best fit to the training data and a smoothness of the fit is found, i.e. minimum overall error  E, see Zaknich (2003). 2.4 Momentum During training, the problem of widely differing eigenvalues often occurs. One technique to deal with this inconvenience is to add momentum γto the weight update formula. This limits the oscillations and adds inertia to the movement through weight space. As may be found in Bishop and Bishop (2023), the effect of the momentum is increasing the effective learning rate along the curve in the weight space, where the gradient remains unchanged. On the contrary, in high curvature regions where the gradient changes significantly, the contribution of the momentum is suppressed and has little to no effect on the effective learning rate value. Fig. 1. Difference between the gradient descent with the momentum term on the left side with faster convergence towards the optimum in comparison to gradient descent without the momentum term oscillating along the path on the right side. 2.5 Mini-Batch In its principle, SGD uses only one data point to calculate gradient estimation of the error function leading to a very noisy estimate, while the computation on the entire data set yields accurate gradient results. However, to determine gradient estimation for each data point is computationally demanding, hence a mini-batch—a small subset of data points—is used to evaluate the gradient at each iteration. According to Bishop and Bishop (2023), an important factor to bear in mind when using mini-batches is that constituent data points should be chosen randomly from the training set. This is due to possible correlations between consecutive data points arising from the way data was collected or stored, for example, it is sorted in an alphabetical or chronological order. 3. NET STRUCTURE A great diversity of net structures can be observed across their applications as a consequence of different classification problems having data sets that differ in their format, and thus, requiring a corresponding network architecture. In this paper we focus on applying CNNs in an atypical setting: designing a classifier using a small data set of 290 samples composed of a low resolution images, see Fig 2. After careful consideration of the dataset’s properties, a simpler structure with a considerably smaller number of Michaela Mesárošová et al. / IFAC PapersOnLine 58-9 (2024) 299–304 301 running average of the magnitudes of recent gradients for the corresponding weight. vℓ=β2vℓ−1+ (1 −β2)[∇E(θℓ)]2(2) θℓ+1 =θℓ−α∇E(θℓ) √vℓ+ϵ,(3) where vℓis the moving average, β2is the squared gradient decay factor of the moving average, and ϵis a small constant to avoid division by zero, for more detail see Tieleman and Hinton (2012). As stated by Kingma and Ba (2015), RMSProp is suited for online and non-stationary settings. Adaptive moment estimation, also called Adam, was first introduced in Kingma and Ba (2015). It is an efficient stochastic optimization that requires first-order gradients. Individual adaptive learning rates are again maintained for all parameters separately. Updates are directly estimated by using a running average of the first and the second moments of the gradients. Moving averages are calculated as follows: mℓ=β1mℓ−1+ (1 −β1)∇E(θℓ) (4) vℓ=β2vℓ−1+ (1 −β2)[∇E(θℓ)]2(5) mℓ=mℓ/(1 −βℓ 1) (6) vℓ=vℓ/(1 −βℓ 2) (7) where β1and β2are the exponential decay rates for the moment estimates. The network parameters then are updated as θℓ+1 =θℓ−αmℓ √vℓ+ϵ(8) If gradients are similar throughout many iterations updates of the weights are able to gain momentum in a certain direction by using a moving average of the gradient. The Adam algorithm is suitable for working with large amounts of data for its small memory requirements. 2.2 Learning rate Learning rate, also referred to as step size, determines the rate of change of the weights. The initial learning rate has a significant impact on the learning speed and the overall training development. When the learning rate is too large, the training error may inadvertently increase rather than decrease, yet with a value too small, training is substantially slower and may become stuck with a high training error, for more detail see Goodfellow et al. (2016). Further remarks from Bengio (2012) and Reed and Marks (1999) identify the learning rate as the most important hyperparameter. Its typical default value is set to 0.01. However, additional tuning is always recommended. 2.3 Regularization A reasonable approach to problem-solving would be to adapt the model complexity according to the complexity of the problem. One way to control generalization and avoid the overfitting phenomenon is by regularization. To include regularization in model training, a penalty function is added to the loss function E(θ), discouraging the parameters from having large magnitudes.  E(θ)=E(θ)+λΩ (9) where Ω is the regularization function and λis the multiplicative parameter. An optimal network is one in which a compromise between the best fit to the training data and a smoothness of the fit is found, i.e. minimum overall error  E, see Zaknich (2003). 2.4 Momentum During training, the problem of widely differing eigenvalues often occurs. One technique to deal with this inconvenience is to add momentum γto the weight update formula. This limits the oscillations and adds inertia to the movement through weight space. As may be found in Bishop and Bishop (2023), the effect of the momentum is increasing the effective learning rate along the curve in the weight space, where the gradient remains unchanged. On the contrary, in high curvature regions where the gradient changes significantly, the contribution of the momentum is suppressed and has little to no effect on the effective learning rate value. Fig. 1. Difference between the gradient descent with the momentum term on the left side with faster convergence towards the optimum in comparison to gradient descent without the momentum term oscillating along the path on the right side. 2.5 Mini-Batch In its principle, SGD uses only one data point to calculate gradient estimation of the error function leading to a very noisy estimate, while the computation on the entire data set yields accurate gradient results. However, to determine gradient estimation for each data point is computationally demanding, hence a mini-batch—a small subset of data points—is used to evaluate the gradient at each iteration. According to Bishop and Bishop (2023), an important factor to bear in mind when using mini-batches is that constituent data points should be chosen randomly from the training set. This is due to possible correlations between consecutive data points arising from the way data was collected or stored, for example, it is sorted in an alphabetical or chronological order. 3. NET STRUCTURE A great diversity of net structures can be observed across their applications as a consequence of different classification problems having data sets that differ in their format, and thus, requiring a corresponding network architecture. In this paper we focus on applying CNNs in an atypical setting: designing a classifier using a small data set of 290 samples composed of a low resolution images, see Fig 2. After careful consideration of the dataset’s properties, a simpler structure with a considerably smaller number of learning parameters was selected. This decision is based on the generally observed fact, that larger CNNs tend to overfit the model with such a small amount of available data. One approach to handle this issue stated by Keshari et al. (2018) is to reduce the number of learnable parameters. 3.1 Convolutional layer One of the first structural properties we consider are convolutional layers, number of kernels and their size. Convolutional layer extracts information or features from the input signal using convolution filters. Given the low resolution of the images in the data set, the commonly used filter sizes sensible for this application are 3 ×3 and 5×5. Hence, we can describe the convolutional layers of the network as a tuple (l1,l 2), where li×liis the size of the convolution filter in i-th layer, for i∈{1,2}. These parameter combinations were tested: (l1,l 2); l1,l 2∈{3,5}.(10) 3.2 Pooling layer Pooling is responsible for downsizing the spatial size of the output from the previous layer by computing an average value in the filter window—average pool—or the popular approach of picking the maximal value in the filter region—max pool —see in Zhou and Chellappa (1988). This reduces computational complexity and allows the subsequent convolutional layer to extract features at a different scale. Similarly to convolutional layers we can describe our selection of pooling layers. If we take into account the image resolution and the fact that key features of the lying position are often concentrated on smaller surfaces, it will also be appropriate to adjust the filter dimensions accordingly. Hence the contemplated sizes were 2×2and3×3 with two possible step sizes 1 and 2. The individual configurations compared in the experiments can be then described by a parameter vector (p1,s 1,p 2,s 2) where the dimension of the i-th pooling layer is pi×piand siis the step size of the i-th filter for i∈{1,2}. Tested combinations were (p1,s 1,p 2,s 2)∈({2,3}×{1,2})2.(11) 2 4 6 8 10 5 10 15 20 25 30 2 4 6 8 10 5 10 15 20 25 30 2 4 6 8 10 5 10 15 20 25 30 2 4 6 8 10 5 10 15 20 25 30 Fig. 2. Data set example of one subject in four lying positions. The chosen pooling method is the max pool. 2The simple reasoning behind this choice is to maximize the pressure indications in key areas such as shoulders, hips, knees and heels that are important for a successful classification. Using the average pool would blur these areas, causing a decreased differentiability between the features of classification classes. 3.3 Activation function The activation function defines the output of a node or a neuron for a given input. It can be thought of as an evaluation of whether or not a neuron should be activated upon the arrival of a specific input. Networks are capable of solving nontrivial problems when nonlinear activation functions are included in the structure, see Hinkelmann (2018). Some of the traditional and modern activation functions with their applications are the sigmoid function, hyperbolic tangent, binary step introduced in McCulloch and Pitts (1943), Rectified Linear Unit (ReLU) described in Nair and Hinton (2010), Leaky ReLU described in Maas (2013), Exponential Linear Unit (ELU) described in Clevert et al. (2016), to name a few. Sigmoid, hyperbolic tangent, ReLU and ELU activation functions were used in the presented experiments. 4. DATA For training purposes pressure map images were used. These maps represent the values measured by the pressuresensitive mattress with a person situated in lying positions, see Fig. 2. The data set consists of 290 images of size 30 ×11. Each image has an associated subject and class number. All classes 1–4 illustrated in Fig. 2 correspond to these positions: •on the back (28.2%), •on the right side (23.7%), •on the left side (21.7%), •on the stomach (26.2%), respectively. Numbers represent the percentage representation of a class in the dataset. Within each of the positions, slight variations may occur such as different positioning of the arms and legs or location on the mattress, i.e., lying in the middle, on the edge or diagonally. The total number of measured subjects is 18. Images are stored as matrices that contain values from 0 to 1, where 0 represents the maximum pressure and 1 is no pressure on the mattress. 5. DESIGN AND PERFORMANCE Architectures were trained using MATLAB 2023 Deep Learning Toolbox. It is necessary to state that the results presented in this paper are only optimal for parameter space earmarked in Sections 3 and 5. There may exist net structures that would achieve similar or better results, however, it is not possible to search the whole parameter space to find the globally best-performing CNN. Our results are quantified using the Accuracy measure (Acc) obtained by cross-validation with 18 folds (number of subjects). 5.1 Structure Before we are able to tune the network’s parameters, the network structure needs to be established. The structural 302 Michaela Mesárošová et al. / IFAC PapersOnLine 58-9 (2024) 299–304 Fig. 3. Dependence of accuracy on the filter configuration and regularization. Convolution filter sizes were set to (l1,l 2)=(3,3) with stride 1. starting point was set to two convolutional layers between which the pooling layers were placed. Three to ten filters were tested in each convolutional layer for filter sizes stated in (10). As for the parameters, the only one resolved at the beginning is the optimization algorithm. After consideration of algorithm properties, Adam was chosen as it combines the advantages of the gradient descent optimization and RMSProp. The learning rate was set to 0.05 for all training scenarios and the range of regularization values is from 0.01 to 7 ·10−5. Other training options remain at default values until the final structure is chosen. Figs. 3 and 4 depict two examples of filter size combinations. Fig. 3 shows that decreasing regularization creates a rather unstable and fluctuating surface accompanied by descending average Acc. For this reason, further tests did not include regularization with values 10−4and 7 ·10−5. None of the combinations shown in Fig. 4 were able to surpass the best results for filter sizes 3 ×3, which is also the case for the remaining filter size variations. Therefore, the final structure of the convolutional layers is six filters in the first layer and 7 filters in the second layer. Weights in the filters were initialized from the normal distribution with zero mean and standard deviation 0.01. The next parameter analysed was the activation function. The experiment included sigmoid, hyperbolic tangent, leaky ReLU and ELU. After evaluation of various combinations, we concluded that the activation function Fig. 4. Dependence of Acc on filter configuration and regularization. Convolution filter sizes were set to (l1,l 2)=(5,3) with stride 1. should be consistent throughout the network. The first rejected activation was sigmoid for its poor results, only around 25% Acc, which is comparable with random guessing. Moreover, it is computationally demanding. Although hyperbolic tangent is also a complex function to compute, it yields the best results closely followed by ReLU and ELU activation functions with cross-validation Acc ≈90%. The final examined structural aspects are the pooling layers. For the reasons mentioned in Section 3 max pooling method was applied in configuration (11). Differences in the classification accuracy of the different configurations varied only slightly (±2%). With smaller sliding steps the accuracy increased, however, lowering the step size escalates the time required for training. If stride 1 is used even in one layer, the training time almost doubles. The size of the used dataset implies that such an increase in the training time still does not present a problem, but in the case of different datasets (number of measurements or data type), it is a factor that needs to be taken into account. Configuration (p1,s 1,p 2,s 2)=(2,1,2,2) achieved the highest accuracy. After the examination of the partial success rates final network structure was selected: •6 convolution filter 3 ×3, •hyperbolic tangent, •max pool 2 ×2, •7 convolution filter 3 ×3, •hyperbolic tangent, •max pool 2 ×2, •fully connected layer, •softmax with 4 classes. This network contains only 1,700 learnable parameters, which is a notably smaller number in comparison with Michaela Mesárošová et al. / IFAC PapersOnLine 58-9 (2024) 299–304 303 Fig. 3. Dependence of accuracy on the filter configuration and regularization. Convolution filter sizes were set to (l1,l 2)=(3,3) with stride 1. starting point was set to two convolutional layers between which the pooling layers were placed. Three to ten filters were tested in each convolutional layer for filter sizes stated in (10). As for the parameters, the only one resolved at the beginning is the optimization algorithm. After consideration of algorithm properties, Adam was chosen as it combines the advantages of the gradient descent optimization and RMSProp. The learning rate was set to 0.05 for all training scenarios and the range of regularization values is from 0.01 to 7 ·10−5. Other training options remain at default values until the final structure is chosen. Figs. 3 and 4 depict two examples of filter size combinations. Fig. 3 shows that decreasing regularization creates a rather unstable and fluctuating surface accompanied by descending average Acc. For this reason, further tests did not include regularization with values 10−4and 7 ·10−5. None of the combinations shown in Fig. 4 were able to surpass the best results for filter sizes 3 ×3, which is also the case for the remaining filter size variations. Therefore, the final structure of the convolutional layers is six filters in the first layer and 7 filters in the second layer. Weights in the filters were initialized from the normal distribution with zero mean and standard deviation 0.01. The next parameter analysed was the activation function. The experiment included sigmoid, hyperbolic tangent, leaky ReLU and ELU. After evaluation of various combinations, we concluded that the activation function Fig. 4. Dependence of Acc on filter configuration and regularization. Convolution filter sizes were set to (l1,l 2)=(5,3) with stride 1. should be consistent throughout the network. The first rejected activation was sigmoid for its poor results, only around 25% Acc, which is comparable with random guessing. Moreover, it is computationally demanding. Although hyperbolic tangent is also a complex function to compute, it yields the best results closely followed by ReLU and ELU activation functions with cross-validation Acc ≈90%. The final examined structural aspects are the pooling layers. For the reasons mentioned in Section 3 max pooling method was applied in configuration (11). Differences in the classification accuracy of the different configurations varied only slightly (±2%). With smaller sliding steps the accuracy increased, however, lowering the step size escalates the time required for training. If stride 1 is used even in one layer, the training time almost doubles. The size of the used dataset implies that such an increase in the training time still does not present a problem, but in the case of different datasets (number of measurements or data type), it is a factor that needs to be taken into account. Configuration (p1,s 1,p 2,s 2)=(2,1,2,2) achieved the highest accuracy. After the examination of the partial success rates final network structure was selected: •6 convolution filter 3 ×3, •hyperbolic tangent, •max pool 2 ×2, •7 convolution filter 3 ×3, •hyperbolic tangent, •max pool 2 ×2, •fully connected layer, •softmax with 4 classes. This network contains only 1,700 learnable parameters, which is a notably smaller number in comparison with 1 Input 6 conv + tanh 6 max pool 6 7 conv + tanh 42 max pool 42 fully conn. Soft max Output Fig. 5. The resulting CNN structure. Bigger blocks represent output from the previous layer with newly obtained dimensions while small boxes illustrate filters with their number of channels and functionality. Fig. 6. Success rates, portrayed by color bar, based on the tuned hyperparameters for mini-batch size 128. usually used CNNs for image processing. For visualization of the final structure see Fig. 5. 5.2 Fine tuning To attain optimal results it is essential to pay attention to fine tuning. For a clear summary of the tuned parameters and their range refer to Table 1. For each parameter, the default value and its surrounding interval were tested. For instance, in mini-batch size powers of two are commonly used to optimize the work with memory, except the last value, which is the whole size of a training set. Had the value been set to 256, the training set would have been passed empty, as the algorithm automatically forms batches only if enough data is available. In Table 1, the influence of the parameters on the Acc decreases from left to right. Table 1. Overview of tuned hyperparameters Learn rate Regularization Epochs Mini-batch Gradient decay 0.05 0.001 200 32 0.9 0.035 0.0002 700 64 0.8 0.02 0.0007 2000 128 0.7 0.01 4000 247 0.005 304 Michaela Mesárošová et al. / IFAC PapersOnLine 58-9 (2024) 299–304 Fig. 6 shows that the majority of results are in the range of 75–85%. This is also the case for the mini-batch of size 247. The best-performing configuration reached 93.1% with a learning rate of 0.005, regularization of 0.001, gradient decay of 0.8 and mini-batch size of 247 throughout 4,000 epochs. The fourth plot in Fig. 6 is overall the one with the highest cross-validation Acc in the whole parameter space ranging from nearly 80–93%, while the rest epoch combinations go as low as 35%. Another perceptible phenomenon is the fact stated in Section 2 that the learning rate has the largest impact on the classification results—the Acc values are mostly changing along the learning rate axis. The second most influential parameter is regularization and in combination with the learning rate it is obvious, that their mutual decreasing also worsens the Acc. Hence it is important to note that with a small learning rate and rather low regularization the Acc value always significantly drops, due to insufficient training or excessively complex models containing high weight values. If we compare the Acc that differ only in their gradient decay we observe that they are practically the same, implying that gradient decay has a little effect on the resulting Acc. For future experiments, this parameter may be left out until more effectual settings are established. It may be tested as one of the last tuning options. The performance of the proposed network has been compared to the results obtained by transfer learning employing GoogLeNet, ResNet and Squeezenet. In all cases, the proposed small-scale network performed better achieving higher accuracy. This confirms the consideration from Section 3 that a simpler network is better suited for this type of task. Considerable difference also lies in the time needed to train the network, where the shallow structure with 1,700 learnable parameters is trained much faster in comparison to transfer learning. 6. CONCLUSION This paper discusses the possibilities of training a small convolutional neural network while having available only a small data set. Although CNNs typically need thousands of data samples to operate reliably or to provide satisfactory accuracy, we have been able to find an appropriate network structure design and discover corresponding optimal parameters through extensive fine-tuning experiments. Learning rate and regularization have shown the strongest impact on the overall accuracy. Number of epochs also has a strong effect. However, this affects only whether the model had enough time to be trained or if the model starts to over-fit on given data. For this reason, it is apt to observe the training process in the early stages of design. Mini-batch has shown a moderate influence and gradient descent had only a minor influence on the final accuracy. Therefore, these two hyperparameters could be tuned at the end of the network design. With the highest reached accuracy being 93.1%, we have shown the applicability of CNNs to this type of classification problem. As a part of future work, the influence of other parameters such as learning rate scheduling or squared gradient decay factor could be examined. Additional expansion of the parameter space such as the number of filters used in convolutional layers may also unveil network structure with improved classification performance. ACKNOWLEDGEMENTS The completion of this paper was made possible by the grant No. FEKT-S-23-8451 – “Research on advanced methods and technologies in cybernetics, robotics, artificial intelligence, automation and measurement” financially supported by the Internal science fund of Brno University of Technology. REFERENCES Beale, M.H., Hagan, M.T., and Demuth, H.B. (2023). Deep learning toolbox reference. Bengio, Y. (2012). Practical recommendations for gradient-based training of deep architectures. CoRR, abs/1206.5533. URL http://arxiv.org/abs/1206.5533. Bishop, C.M. and Bishop, H. (2023). Deep Learning: Foundations and Concepts. Springer, Cham, 1st ed. edition. Clevert, D.A., Unterthiner, T., and Hochreiter, S. (2016). Fast and accurate deep network learning by exponential linear units (elus). Ferreyra-Ramirez, A., Aviles-Cruz, C., RodriguezMartinez, E., Villegas-Cortez, J., and Zu˜niga-Lopez, A. (2019). An improved convolutional neural network architecture for image classification. In Lecture Notes in Computer Science, volume 11524, 89–101. Springer International Publishing, Cham. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. The MIT Press. Hinkelmann, K. (2018). Neural networks, p. 7. Keshari, R., Vatsa, M., Singh, R., and Noore, A. (2018). Learning structure and strength of cnn filters for small sample size training. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9349–9358. doi:10.1109/CVPR.2018.00974. Kingma, D.P. and Ba, J. (2015). Adam: A method for stochastic optimization. URL http://arxiv.org/abs/1412.6980. Maas, A.L. (2013). Rectifier nonlinearities improve neural network acoustic models. McCulloch, W.S. and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biology, 5, 115–133. Nair, V. and Hinton, G.E. (2010). Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning, 807–814. Reed, R. and Marks, R.J. (1999). Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks. The MIT Press. doi: 10.7551/mitpress/4937.001.0001. Tieleman, T. and Hinton, G. (2012). Lecture 6.5-rmsprop, coursera: Neural networks for machine learning. University of Toronto, Technical Report. Zaknich, A. (2003). Neural networks for intelligent signal processing. World Scientific, New Jersey. Zhou, Y.T. and Chellappa, R. (1988). Computation of optical flow using a neural network. IEEE 1988 International Conference on Neural Networks, 2, 71–78. doi:10.1109/ICNN.1988.23914.