Backpropagation and Training Models in Neural Networks and Deep Learning

978-93-7143-144-6 483 Chapter – 29 Backpropagation and Training Models in Neural Networks and Deep Learning Mr. Amit Kumar Ojha Tech Lead (Application Development) Org: Tmotions Global Pvt. Ltd. E-mail: [email protected]om Ms. Nimisha Tiwari Assistant Professor (Department of ECE) GGSIPU/JIMSEMTC E-mail: nimishaatiwaa[email protected] Ms. Pallavi Ashok Kulkarni Assistant Professor (Department of Electronics and Communication Engineering) Angadi Institute of technology and management, Belagavi, Karnataka Email id - [email protected] Ms. Nitika Assistant Professor (Department of Artificial Intelligence and Data Science) College - IIMT College of Engineering, Greater Noida Email id- [email protected] Abstract:-The fundamental method that transformed artificial neural network training and made possible the deep learning revolution that has revolutionised artificial intelligence in the twenty-first century is backpropagation. The backpropagation algorithm and the larger field of neural Mr. Amit Kumar Ojha……etc. 484 network development techniques are thoroughly examined in this chapter. We start with the theoretical underpinnings, utilising the chain rule of calculus to derive backpropagation from fundamental principles. We next show that this sophisticated mathematical framework makes it possible to compute gradients in multi-layer networks efficiently. Stochastic gradient descent and its contemporary variations, such as Adam, RMSprop, and AdaGrad, are among the basic optimisation algorithms covered in this chapter. There algebraic traits and real-world efficiency characteristics are examined. We go over important components of the procedure for training, such as batch normalisation approaches, regularisation tactics to avoid excessive fitting, activated function choosing, and weight initialisation procedures. Gradient clipping, learning rate scheduling, momentum-based optimisation, and novel learning rate techniques are among the advanced subjects discussed. We address frequent issues like vanishing and exploding gradients, optimise computing performance, and offer insights into debugging training processes throughout the chapter, emphasising both knowledge of theory and concrete application considerations. By bridging traditional machine learning theory with modern deep learning practice, the content gives users the skills they need to create, deploy, and diagnose complex neural network education streams. Keywords:-Backpropagation, Gradient Descent, Neural Network Training, Deep Learning Optimization, Stochastic Gradient Descent, Adam Optimizer, Weight Initialization, Regularization, Batch Normalization, Learning Rate Backpropagation……….etc. 485 Scheduling, Vanishing Gradients, Exploding Gradients, Momentum, AdaGrad, RMSprop, Activation Functions, Loss Functions, Overfitting Prevention, Gradient Clipping, Neural Network Architectures Introduction:-An important turning point in the history of artificial intelligence was the emergence with backpropagation in the 1980s. Although neural networks were first proposed decades ago, their practical use was severely constrained by the absence of effective training techniques. The algorithmic innovation that made training multi-layer neural networks possible was backpropagation, which was independently rediscovered by a number of scientists including the aforementioned Hinton, and Williams in 1986. Backpropagation, which allows for the effective computation of gradients of a loss coefficient with respect to all parameters in the neural network, is fundamentally a consequence of the mathematical chain rule. Because current deep learning models include thousands or millions of parameters and determining gradients blindly would be economically prohibitively costly, such efficacy is essential. By computing partial derivatives layer by layer and reusing intermediate computations, the approach sends error signals backwards across the structure of the network. There is much more to training neural networks than just using backpropagation. Network architecture, loss function selection, optimisation methods, regularisation techniques, and tuning of hyperparameter are just a few of the intricate Mr. Amit Kumar Ojha……etc. 486 design choices that practitioners must negotiate. Each of these components interacts in nuanced ways, and both theoretical knowledge and experience-based practical intuition are necessary for effective training. This chapter is organised to gradually increase comprehension, beginning with the fundamentals of mathematics and progressing to complex contemporary methods. We focus not just on the mechanics of algorithms but also on the fundamental ideas that underpin their creation and the pragmatic factors that influence how well they work in practical settings. Mathematical Foundations of Backpropagation Neural Network Architecture and Forward Propagation:- Layers of connected nodes, or neurones, make up a neural network that moves forward. The activation of neurone j in layer l of a network with L layers is represented as a(l,j). There are two steps involved in moving from one of the layers to the next: A function of activation that is nonlinear after a linear modification.A weighted total of the inputs plus a bias term is calculated by the linear transformation. This is represented for layer l as z(l,j) = Σ w(l,j,k) * a(l-1,k) + b(l,j), where b(l,j) is the bias ingredient and w(l,j,k) is the weight that connects neurone k in layer l-1 to neurone j in layer l. In matrix notation, this can be expressed more succinctly as z(l) = W(l) * a(l-1) + b(l). Nonlinearity is introduced by the activation function and is necessary for a neural network to learn intricate patterns. The sigmoid function σ(z) = 1/(1 + e^(-z)), the hyperbolic tangent tanh(z), and the rectangle-shaped linear unit (ReLU), which is defined as ReLU(z) = max(0, z), are examples of common Backpropagation……….etc. 487 function activation. Both the network's representational capability and training dynamics are greatly impacted by the activation function selection. The technique of calculating the network's output given an input is known as forward propagation. Eventually compute z(l) and a(l) repeatedly for each succeeding layer until we reach the output layer L, beginning with the input layer (layer 0), where a(0) = x (the input vector). The network's prediction is represented by the most recent activation, a(L). Loss Functions and Training Objectives:-The difference between the algorithm's predictions and the actual target values is measured by the loss function. The loss L(y, ) for a single training case (x, y), where x is the input variable and y is the target, indicates how well the prediction ⷽ = a(L) fits y. Training behaviour is significantly impacted by the loss function selection, which is task-specific. The mean squared error (MSE) is frequently utilised for regression tasks: L(y, ⷿ) = 1/2 * ||y - ⷿ||². In derivative calculations, the factor of 1/2 is added for mathematical ease. When sigmoid activation is used in the output layer for binary classification, the conventional binary cross-entropy loss is L(y, ⷽ) = -[y log(ⷿ) + (1-y) log(1-ⷷ)]. The categorical cross-entropy loss technique in conjunction with a softmax activation function in the output layer is commonly used for multi-class classification. Softmax(z_i) = e^(z_i) / Σ e^(z_k) is the function that transforms raw output Mr. Amit Kumar Ojha……etc. 488 scores (logits) into a probability distribution. Then, if y is usually one-hot encoded, the cross-entropy loss is L(y, ⁿ) = -₂ y_i log(⿿_i). The ultimate goal of training is to reduce the average loss across all training examples, or the empirical risk: J(W, b) = (1/N) Σ L(y^(i), f(x^(i); W, b)), where f is the neural network function parameterised by weighting W and prejudices b, and N is the total amount of training examples. How to Derive the Backpropagation Algorithm:-The gradient of the loss function with respect to each network parameter is calculated using backpropagation. The crucial realisation is that by employing the calculus chain rule in a certain order and reusing previous processes, we may compute these variations effectively. For neurone j in layer l, we define the error term δ(l,j) = ∂L/∂z(l,j). This shows the degree to which the loss deviates from the pre-activation value. This is simple for the output layer L: δ(L,j) = ∂L/∂a(L,j) * Òa(L,j)/∂z(L,j) = ØL/∂a(L,j) * σ'(z(L,j)), wherein σ' is the activated function's derivative. We use the chain rule recursively for hidden layers. The faults in layer l+1 determine the error at layer l: δ(l,j) = [Σ w(l+1,k,j) * δ(l+1,k)] * σ'(z(l,j)). This formula, which is weighted by the connection strengths, illustrates how errors spread backward across the network. Backpropagation……….etc. 489 The gradients with regard to the biases and weights follow immediately after we have calculated the error terms for each layer. ∂L/∂w(l,j,k) = δ(l,j) * a(l-1,k) is the gradient with respect to a weight, and ØL/∂b(l,j) = δ(l,j) is the gradient with regard to a bias. Efficient adjustments to parameters are made possible by these elegant formulations. Two passes are required to finish the backpropagation method. Every activation is calculated and stored during the forward pass. The backward pass uses the stored activations to calculate weight and bias gradients while computing error terms beginning at the output layer and moving backward. The technique is extremely efficient because of this multiplepass structure, which guarantees that the gradient of each parameter is calculated in time proportionate to the overall amount of connections. Gradient Descent and Optimization Algorithms Gradient Descent in Batch:-An iterative optimisation technique called gradient descent shifts parameters in the direction of the loss function's steepest descent. The update rule is θ(t+1) = θ(t) - η * ΨJ(θ(t)), where θ is the gradient of the loss function, η is their understanding rate, and θ represents all parameters. Before performing a single update, batch gradient descent uses the whole training dataset to compute the gradient. This guarantees that every update proceeds in the direction that minimises the training loss the greatest. However, calculating the entire gradient requires a lot of memory and work for Mr. Amit Kumar Ojha……etc. 490 large datasets. Batch descents with gradients can also get stuck in saddle spots or regional minima. The rate of learning A crucial hyperparameter that regulates the step size is η. The method might overshoot the minimum and diverge if η is too big. Convergence will be very slow if η is too small, possibly requiring a great number of repetitions. It frequently takes trial and error to get the right learning pace. Descent of a Stochastic Gradient:-By calculating the gradient using just one training example or a tiny mini-batch at every run, randomly generated gradient descent (SGD) overcomes the computational constraints of batch gradient descent. With (x(i), y(i)) being a randomly chosen training circumstance, the amended rule becomes θ(t+1) = θ(t) - η * ΨL(θ(t); x(i), y(i)). Individual SGD updates are computationally faster than batch updates, which permits more frequent parameter updates even though they are noisier. Although additional rounds can be needed, this higher update frequency frequently results in faster convergence in terms of wall-clock time. Additionally, the stochasticity aids in escaping shallow local minima and has a positive regularisation effect. A workable compromise is mini-batch SGD, which computes gradients over tiny batches of instances (usually 32, 64, 128, or 256). This method preserves most of the computational efficiency of pure SGD while taking advantage of vectorised operations on contemporary technology. Another hyper Backpropagation……….etc. 491 parameter that influences model operation and learning speed is the small batch size. Optimisation Based on Momentum:-By building up an exponentially declining average move of previous gradients, momentum algorithms speed up gradient descent. The update rule becomes v(t+1) = β * v(t) + η * ΨJ(θ(t)) and θ(t+1) = θ(t) - v(t+1), where β is the momentum coefficient, usually set at 0.9, and v is the velocity vector. Momentum dampens oscillations in directions where the gradient frequently changes sign while accelerating progress in directions where the gradient consistently points in the same direction. In situations where the lost landscape resembles a tight ravine or has considerable curvature, where normal downward gradients would bounce over walls, this is especially advantageous.By calculating the gradient at a lookahead position, Nesterov Accelerated Gradient (NAG) enhances standard momentum. With improved convergence qualities, particularly close to the optimum, the update becomes v(t+1) = β * v(t) + η * ΨJ(θ(t) - β * v(t)). The algorithm can more efficiently slow down ahead exceeding the limit thanks to this lookahead approach. Four Responsive Instruction Rate Techniques:-Using historical gradient data, adaptive learning rate techniques automatically modify the learning rate for every parameter. AdaGrad modifies each parameter's learning rate in an inverse relationship to the cubic root of the total of all Mr. Amit Kumar Ojha……etc. 498 Since its introduction by Ioffe and Szegedy in 2015, batch normalisation has grown to be an essential part of contemporary deep learning architectures. It tackles the issue of internal covariate shift, in which the transportation of stage inputs shifts as prior layers' parameters are updated during training. This change makes training more difficult, necessitating cautious initialisation and lower learning speeds. Each layer's activations are normalised by batch normalisation so that every small batch has a zero mean and unit variance. Batch normalisation for a layer with ddimensional input x = (x(1),..., x(d)) modifies each dimension separately: xρ(k) = (x(k) - μ_B(k)) / √(σ²_B(k) + ε), where ε is a tiny constant for numerical robustness and μ_B and σ²_B are the mini-batch mean and volatility. Batch normalisation adds learnable scale as well as shift parameters to maintain the network's representational power: y(k) = γ(k) * x(k) + β(k). If it turns out to be the best option, these factors enable the network to reverse the normalisation. Backpropagation is used to learn weights along with the scale characteristics γ and move parameters β. Backpropagation……….etc. 499 Addressing Gradient Challenges The Vanishing Gradient Problem:- When gradients grow incredibly small as they propagate backward through numerous layers, learning in early levels is effectively prevented. This phenomenon is known as the vanishing gradient problem. Because the derivatives of sigmoid and tanh activation functions are minimal for large positive or negative inputs, this issue is especially serious. The gradient in a deep network is the product of numerous terms; if these terms are smaller than one, their product diminishes rapidly with thickness. Vanishing gradients are mitigated by a number of methods. Because ReLU prevents gradient decay by having a continuous gradient of one for positive inputs, using ReLU activation functions is beneficial. Gradients don't disappear or blow up during the early phases of training thanks to careful weight initialisation, especially He initialisation for ReLU networks. By avoiding internal correlation shift, batch normalisation stabilises gradient flow. Gradients can move straight through the network without going through several nonlinear transformations thanks to residual connections, which were introduced in ResNet Mr. Amit Kumar Ojha……etc. 500 architectures. When x is the input, the residual block computes F(x) + x instead of just F(x). Networks of hundreds or even thousands of layers can now be trained because to this architecture. The Exploding Gradient Problem:- Gradients that grow exponentially big during backpropagation are known as "exploding gradients," which can result in divergent training and numerical instability. Because gradients can develop exponentially with sequence length, this issue is especially prevalent in recurrent neural networks processing lengthy sequences. A straightforward and efficient method for dealing with exploding gradients is gradient trimming. Gradients are rescaled when their norm above a threshold: if ||g|| > threshold, set g ÷ (threshold / ||g||) * g. This keeps the gradient direction intact while preventing any one gradient update from being excessively huge. The range of common threshold values is 1 to 10. LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) architectures offer gating features that aid in controlling gradient flow over time for recurrent networks. In order to provide more robust training for lengthy sequences, these architectures incorporate multiplicative gates that are capable of learning when to transmit gradients as well as how to block them. Backpropagation……….etc. 501 Scheduling and Adaptation of Learning Rates:-One of the most crucial hyperparameters in neural network training is the learning rate. Seldom is a constant learning rate ideal for training; in the early stages, higher learning rates allow for quick advancement, but in the latter stages, lower acquisition rates are required to optimise parameters and reach a good solution. During the training, learning rate programming dynamically modifies the process of learning rate. At predefined epochs, step decay lowers the learning rate by a factor (usually 0.1). For instance, in a 100-epoch training run, the learning rate might be divided by ten at epochs 30, 60, and 90. This straightforward method works well, but setting the schedule correctly takes domain expertise. The learning rate is continuously reduced by exponential decay, as η_t = η_0 * e^(-kt), where k is a decay constant. The formula for linear decay is η_t = η_0 * (1 - t/T)^p, where p is a power parameter and T is the total number of training steps. These seamless timetables prevent abrupt shifts in the mechanics of learning. The learning rate is adjusted by cosine annealing using a cosine curve: η_t = η_min + (η_max - η_min) * (1 + cos(πt/T)) / 2. This allows the optimisation to explore several local minima and offers smooth decay with the ability to reset the schedule on a regular basis. Warm restarts combine frequent resets to the starting learning rate with cosine annealing. Over the first few epochs, the learning rate warm-up technique progressively raises the learning rate from a low Mr. Amit Kumar Ojha……etc. 502 starting value to the base learning rate. In the early phases of training, when the model is far from convergence, this avoids instability. When training highly deep learning networks or using big batch sizes, warm-up is especially crucial. Advanced Training Techniques Transfer Learning and Fine-Tuning:-Transfer learning improves performance on a related task by utilising knowledge gained from one assignment. This usually entails adapting a model that has been pre-trained on a big dataset to a new job with less data in deep learning. This method is now commonplace, especially in computation of natural languages and vision applications. There are three steps in the standard workflow. First, a sizable source dataset is used to train a base model. Second, only the final classification layer is trained on target data, and the weights that were already trained are frozen and utilised as the characteristic extractor for the target job. Third, a relatively low learning rate can be used to fine-tune individual layers or the neural network as a whole on the target dataset. Different levels of abstraction are learnt by different layers. In convolutional networks, task-specific features are learnt by later layers, whereas generic features like edges as well as textures are learnt by early layers. Discriminative fine-tuning, which uses variable learning rates between layers—smaller Backpropagation……….etc. 503 rates for early layers and bigger rates for later layers—is frequently advantageous for fine-tuning. Learning Program and Purchasing Samples:-Educational learning is modelled after human learning, in which ideas are taught in progressively more challenging order. Presenting training examples in a logical sequence, beginning with simpler examples and progressively introducing more challenging ones, is the objective. Better end-user performance and quicker convergence may result from this. Establishing a plan for giving examples and a measure of example complexity are necessary for implementing curricular learning. Loss values, prediction confidence, other specific to the domain criteria may be used to determine difficulty. The curriculum may be preset or flexible, changing according to the model's present performance. Conclusion and Future Directions The incredible success of contemporary deep learning has been made possible by backpropagation and the training techniques based on it. The discipline has created a comprehensive toolkit for efficiently training neural networks, ranging from the complex optimisation and regularisation approaches to the elegant mathematical basis of the backpropagation algorithm. This chapter's concepts serve as the basis for comprehending and applying neural network training. Nonetheless, the sector is still developing quickly. More effective training algorithms, a deeper comprehension of the optimisation landscape, Mr. Amit Kumar Ojha……etc. 504 easier-to-train structures, and methods for training with little labelled data are examples of recent advancements. Improved optimisation algorithms that converge more quickly and reliably, strategies for training larger models more effectively, training methods that require less computation and data, and a better understanding of generalisation and the relationship between training and test performance are some of the future research directions. Numerous design decisions that are currently made by hand could be automated via meta-learning and neural architecture search. Gaining an understanding of these basic ideas offers a strong basis for both making efficient use of the deep learning tools already available and advancing the subject in the future. The significance of principled training approaches only increases when neural networks are scaled to previously unheard-of sizes and used to ever-more complicated challenges. One of the most important advances in artificial intelligence is the evolution from backpropagation to contemporary deep learning, and the tale is still being told. References 1. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533-536. 2. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Backpropagation……….etc. 505 3. Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 4. Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics (pp. 249-256). 5. He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision (pp. 1026-1034). 6. Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning (pp. 448-456). 7. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1), 1929-1958. 8. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778). Mr. Amit Kumar Ojha……etc. 506 9. Loshchilov, I., & Hutter, F. (2017). SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983. 10. Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. 11. Smith, L. N. (2017). Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV) (pp. 464-472). 12. Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning (pp. 41-48). 13. Hochreiter, S., & Schmidhuber, J. (1997). Long shortterm memory. Neural computation, 9(8), 1735-1780. 14. Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) (pp. 807-814). 15. Sutskever, I., Martens, J., Dahl, G., & Hinton, G. (2013). On the importance of initialization and momentum in deep learning. In International conference on machine learning (pp. 1139-1147). 16. Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017). mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412. Backpropagation……….etc. 507 17. Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., & Yoo, Y. (2019). CutMix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6023-6032). 18. Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450. 19. Wu, Y., & He, K. (2018). Group normalization. In Proceedings of the European conference on computer vision (ECCV) (pp. 3-19). 20. Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the difficulty of training recurrent neural networks. In International conference on machine learning (pp. 1310-1318). 21. Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., ... & Wu, H. (2017). Mixed precision training. arXiv preprint arXiv:1710.03740. 22. Chen, T., Xu, B., Zhang, C., & Guestrin, C. (2016). Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. 23. You, Y., Gitman, I., & Ginsburg, B. (2017). Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888. 24. Reddi, S. J., Kale, S., & Kumar, S. (2019). On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237.