Towards a library of deep neural networks for experimenting with on-device training on microcontrollers

Towards a library of deep neural networks for experimenting with on-device training on microcontrollers Soufiane Aatab Computer Architecture Department Universitat Polit` ecnica de Catalunya Barcelona, Spain [email protected] Felix Freitag Computer Architecture Department Universitat Polit` ecnica de Catalunya Barcelona, Spain [email protected] Abstract—Despite the limited computing capacity of microcontrollers, there is a growing interest in training neural networks on these devices for tiny machine learning applications. The traditional approach is to train models off-device on powerful machines, but this limits the model’s ability on the microcontroller to adapt to changing data. A limitation for further research on on-device training is the lack of open source deep neural network implementations for the training on microcontroller boards. In this paper, we present the design and implementation of a library in C++ to enable neural network training and inference on microcontroller boards. We conduct a functional validation of the implementation by comparing a Convolutional Neural Network (CNN) and autoencoder model defined with our library with a reference implementation. We then integrate the library into a machine learning application and flash it on an Arduino Portenta board to demonstrate its usability for on-device training. The developed library is open source and is flexible for defining different neural network architectures. Researchers and developers can use the given implementation as a tool for building new trainable machine learning applications, but can also extend it with recent model optimization techniques for tiny nodes from the research in this field. Index Terms—embedded learning, microcontrollers, deep learning I. INTRODUCTION Embedded learning, also called TinyML, is the evolution of deploying machine learning models on ever smaller computing devices such as microcontrollers and performing the sensor data capture, processing, and machine learning directly on the device [1]. The increasing computing capacities of 32 bit microcontrollers and the availability of free development tools have led to a vibrant community of makers, innovators, and industries around TinyML1. Today’s most used approach for embedded learning is to do the neural network training off-device. The model is first trained on powerful machines or in the cloud, leveraging their storage and computing capacity to train the neural network with large data sets. Then, tools such as Tensorflow Light2 or cloud-based services optimize the model size to fit into 1https://www.tinyml.org/ 2https://www.tensorflow.org/lite/microcontrollers the small memory of the microcontroller board [2]. Hereafter, the model is converted into a C byte array. This byte array is integrated into the application code flashed to the microcontroller. For doing inference with the model, the application code uses the functions provided by the included TensorFlow Lite for Microcontrollers C++ library. The model is stored in the Flash memory of the microcontroller. This approach, however, does not allow doing on-device training to change the model weights. A recent survey on reformable TinyML [3] reviews ondevice training, referring to machine learning model training on the microcontroller itself. Compared to off-device training, there are several limitations. The training of the model on the microcontroller faces the computing resource constraints of the device. The memory of the microcontroller is very small and does not allow to store any large dataset. Furthermore, the computing capacity is small, which increases the training time. Finally, the quantization and pruning steps that are typically performed in off-device training to reduce the model size are not yet consolidated for on-device training [4]. But the opportunity for on-device training is to overcome the staticity of the neural network model on the microcontroller. This is needed, for instance, when an application seeks to adapt models to address concept drift of data or wishes models to be completely re-trained to perform new tasks. While the interest of the community in on-device training is growing, there is only limited availability of open source deep neural network implementations that have been demonstrated in the training on microcontroller boards. Since open source Python-based machine learning frameworks, such as Tensorflow and PyTorch among others, have been shown to be very beneficial for the machine learning community, contributing an open source C++ library for machine learning models suitable for the training on microcontrollers can support the TinyML community to develop new types of tiny machine learning (ML) applications. With this goal in mind, we present in this paper a C++ library implementation which allows to integrate the definition of several neural network architectures for being trained within © 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. https://dx.doi.org/10.1109/WF-IoT58464.2023.10539445 a ML application running on a microcontroller. We demonstrate the usage with a deployment of a machine learning application on an Arduino Portenta board that integrates the library. The main contributions of the paper are: •A flexible C++ library that allows to define and train CNN, autoencoder, and feedforward neural networks in tiny microcontroller boards. •Demonstration of the usage of the library for on-device training by integrating it into a machine learning applications for a microcontroller board. •The software is published in a public git repository and available for usage and extension by the community3. II. RELATED WORK TinyML refers to the application of machine learning in tiny devices, such as microcontroller boards. Applications of TinyML are found, for instance, in remote Internet of Things (IoT) nodes. In [5], a mosquito logger running in a remote IoT node is presented. In that system, sensing, signal processing, and machine learning task are performed directly on the IoT device. A CNN was used which achieved more than 90% accuracy in the task. However, the CNN was trained offdevice. On-device training with the aim of the adaptability of the trained machine learning models is presented in [6]. Specifically, incremental transfer learning for a KNN classifier was proposed. With image and audio benchmarks the authors showed the feasibility of on-device incremental transfer learning for two hardware platforms, a Raspberry Pi 3B+ and an STM32F7 microcontroller with 512 kB of RAM and 2 MB of flash. In [7] the on-device training of an autoencoder model on a microcontroller is presented. The emergence of new patterns over time motivated on-device training. The authors extend the X-CUBE-AI tool from STMicrolectronics to implement the system. This approach leveraged the optimization techniques provided by the tool but restricts the solution to STM32 microcontrollers. The experiments with a DC motor anomaly detection task showed the successful operation of the neural network trained on the device. In TinyOL [8] a pre-trained autoencoder model is posttrained on an Arduino Nano 33 BLE board to classify new data patterns. The authors call their approach incremental online training. The task was anomaly detection in the vibration patterns of a USB fan. The results showed that by on-device training the neural network could adapt to new conditions and its performance improved over time. In the TinyFedTL, an Arduino Nano 33 BLE board was used for an image recognition task [9]. The authors apply federated transfer learning by using a compressed version of TensorFlow’s MobileNet as a starting point. Then they train only the output layer by backpropagation on the device. The approach allowed leveraging a pre-trained CNN-based feature 3https://github.com/SoufianeAatab/TFG-Federated-Learning extractor and training only the fully connected last layer for the specific task on the device. In [10] a feedforward neural network was applied for a keyword spotting task. The neural network was trained on-device on an Arduino Nano 33 BLE supported by federated learning. The size of the neural network could have a maximum of 25 neurons in the hidden layer due to the memory constraints of the microcontroller. With regards to C/C++ libraries for neural networks, the KANN framework [11] and tiny-dnn [12] have open git repositories. KANN is presented as a standalone, lightweight library that offers building neural networks of type Multilayer Perceptron (MLP), CNN and Recurrent Neural Network (RNN). KANN as it is provided compiles on a PC where also the examples run. However, references about the usage of KANN for on-device training on microcontroller boards are lacking. The last commit in the git repository was two years ago. The tiny-dnn project provides various types of layers, activation functions, loss functions, and optimization algorithms, for building and training several types of neural networks. It is presented as a library suitable for embedded systems and IoT devices. The given code compiles on a PC but concrete examples that show the usage of the library for training neural networks on tiny embedded systems are not provided. The last commit in the git repository was made five years ago and the project is indicated as not being maintained anymore. From the reviewed work it can be seen that there is a diversity of models such as autoencoder, CNN, and MLP that are used on microcontrollers. There is also an increasing body of research work that studies on-device training. A few open source C/C++ neural network implementations are available, but without concrete examples for their usage in TinyML applications, and the projects seem not to be actively developed anymore. Finally, on-device training on microcontrollers has not only the potential for being used in standalone devices but can also be a building block for applying federated learning in these IoT devices. III. DESIGN AND IMPLEMENTATION OF THE LIBRARY The aim is to provide neural network building blocks by a C++ library implementation that enables the developer of TinyML applications to flexibly define a machine learning model with the number of layers, type of layers, and hyperparameter configuration. Our goal is inspired by the benefits of the Tensorflow/Keras framework for training neural network models on a PC or in the cloud. But differently, our implementation is aimed to be used as a library integrated into a C++ application that runs on a microcontroller for the training and inference with a machine learning model. A. Data structures Our implementation defines three different data structures named M, M3, M4. The first one named M consists of a matrix data structure, which is used to represent 2D arrays. Additionally, it contains three static methods for initializing matrices with specific values such as ones, zeros, and random values. The data structures M3 and M4 both have the same implementation, but the first one represents 3D arrays and the other 4D arrays. They have several overloaded operators such as the +, -, *, and / operators which perform element-wise addition, subtraction, multiplication, and division respectively. The M data structure is required in the fully connected layer implementation to store the weights and biases. The M3 and M4 data structures are required for the convolutional layer implementation for storing 3D and 4D data. B. Activation functions The library provides a set of pre-defined activation functions that are readily available for use in constructing neural networks. These functions include Sigmoid, Tanh, Relu, and Softmax, along with their respective derivatives, which have been fully implemented. C. Fully connected layer The fully connected layer is represented as a struct named Dense and has member variables that represent the layer’s weights, biases, gradients, and momentum. The layer has functions for forward and backward propagation, called forward and backward, respectively. The forward function takes an input matrix, multiplies it with the layer’s weights, adds the biases, and returns the output matrix. The backward function takes a gradient matrix as input and calculates the gradient of the loss for the input, which is then passed back to the previous layer. D. Conv2D Layer The Conv2D layer performs 2D convolution operation on an input image tensor (in our case M3). The layer is implemented using a structure called Conv2D. The structure contains the input tensor size, output tensor size, number of kernels, and stride value. The forward() method performs the 2D convolution operation on the input image tensor. The convolution is performed by iterating over each output channel and output pixel location and computing the dot product between the kernel and the input tensor. The bias term is added to the convolution output. The result is stored in the output tensor. The backward() method performs the backward propagation of the convolution operation. It computes the gradients for the input tensor, kernel parameters, and bias term using the standard backpropagation equations. E. Max Pooling The max pooling layer is a common operation in convolutional neural networks for downsampling and feature selection. Our implementation contains the forward and backward pass of the layer. The forward() method takes an input tensor Xand then iterates through each channel of X, dividing it into kernelsized blocks and selecting the maximum value from each block. The selected maximum values are stored in the output tensor, and their indices are stored in the indices member variable for use in the backward pass. The backward() method takes the gradient from the subsequent layer and computes the gradient for Xby using the indices stored in the indices member variable to set the gradient of the corresponding block in the input tensor to the gradient of the corresponding block in the output tensor. F. Loss functions Our library includes three different types of loss functions: CrossEntropy, BinaryCrossEntropy, and MSE. These loss functions are used to measure the difference between the predicted values and the actual values in a machine learning model. CrossEntropy is typically used in classification problems with multiple classes. It calculates the difference between the predicted probabilities and the true probabilities of each class. BinaryCrossEntropy is used in binary classification problems, where there are only two possible outcomes. It measures the difference between the predicted probabilities and the true probabilities of the positive class. Mean Squared Error (MSE) is a common loss function used in regression problems. It calculates the average squared difference between the predicted and actual values. G. Hyperparameters Two optimization algorithms for weight updates are provided: stochastic gradient descent (SGD) and mini-batch gradient descent. Both algorithms can be configured with a learning rate and momentum. SGD updates the weights using the gradient of the loss function computed on a single training example, while mini-batch gradient descent updates the weights using the average gradient over a mini-batch of training examples. IV. EXPERIMENTAL VALIDATION A. Methodology We first conduct a functional validation of the C++ library with two neural network architectures. For this, we compare the training and inference performance of our implementation with that of a reference implementation of the same models by the Tensorflow/Keras framework. For the validation of the functional performance, our library is compiled to run on a PC. B. Autoencoder functional validation For this experiment, we define with the library an autoencoder architecture consisting of an input layer of 29 neurons, followed by two hidden layers with 30 and 20 neurons respectively, another hidden layer with 30 neurons, and finally an output layer with 29 neurons. The activation function used in all layers is tanh, except the last one which has none to facilitate the reconstruction of the input. To validate the functionality of our implementation, we use a credit card dataset obtained from Kaggle4, which has 227,451 elements for training and 56,962 for validation. The autoencoder is trained with a batch size of 32, a learning rate of 0.0001, and a momentum of 0.9. 4https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud Figure 1 compares the loss of our implementation with that of Tensorflow/Keras. It can be seen that the evolution of the loss in the training and validation dataset is the same in both implementations. (a) Comparison of the loss of Tensorflow/Keras and our C++ implementation in the training dataset. (b) Comparison of the loss of Tensorflow/Keras and our C++ implementation in the testing dataset. Fig. 1: Autoencoder implementation performance comparison. C. CNN functional validation For this experiment, we define with the library a CNN architecture which consists of one convolutional layer, followed by a max pooling layer, and two dense layers. The hyperparameters used are a learning rate of 0.01 and a momentum of 0.9. For the convolutional layer, we use a sigmoid activation function and 16 kernels with a kernel size of 3x3 and a stride of 1. For the max-pooling layer, we choose a pool size of 3x3 and a stride of 3. The two dense layers have 32 and 10 neurons, respectively, with sigmoid activation for the first dense layer and softmax activation for the second dense layer. The loss function used is cross-entropy. In order to validate our implementation, we use the popular MNIST dataset, with a training set of 60000 and a validation set of 10000, and we do the implementations of the same neural network model in Tensorflow/Keras. Figure 2a shows the comparison of the accuracy evolution of the CNN. Both implementations present a similar evolution in terms of increasing accuracy with a larger amount of training epochs. In Figure 2b the evolution of the loss can be seen. The comparison shows a similar evolution of the loss in both implementations. (a) Comparison of the accuracy of Tensorflow/Keras and our C++ implementation. (b) Comparison of the loss of Tensorflow/Keras and our C++ implementation. Fig. 2: CNN implementation accuracy/loss comparison. D. Defining neural network architectures with the library In this section, we look at how the application developer of a machine learning application can define a neural network when using the library. Principally, the library is contained in a single file named nn.h and has to be included in the C++ main file. Then, in the main file, the neural network is defined. The example presented in Listing 1 shows the autoencoder network used in section IV-B, and Listing 2 corresponds to the CNN that we introduced in section IV-C. It can be seen that the architectural definition of both networks applies the same pattern, consisting of five sections. The data structures of type Layer and Mare used to store the data of the building blocks of the neural network. Listing 1: Snippet autoencoder definition with the C++ library. //Layer creation autoencoder Layer *l1 = Layer::create(29, 30); Layer *l2 = Layer::create(30, 20); Layer *l3 = Layer::create(20, 30); Layer *l4 = Layer::create(30, 29); //Forward M a = Tanh(l1->forward(X[j])); M b = Tanh(l2->forward(a)); M c = Tanh(l3->forward(b)); M o = l4->forward(c); //Backpropagation M _d4 = MsePrime(X_train[j], o); M _d3 = l4->backward(_d4) *TanhPrime(c); M _d2 = l3->backward(_d3) *TanhPrime(b); M _d1 = l2->backward(_d2) *TanhPrime(a); //Update weights l1->dw += l1->getDelta(_d1, X_train[j]); l1->db += _d1; l2->dw += l2->getDelta(_d2, a); l2->db += _d2; l3->dw += l3->getDelta(_d3, b); l3->db += _d3; l4->dw += l4->getDelta(_d4, c); l4->db += _d4; //SGD l1->UpdateWeights(lr, batchsize); l2->UpdateWeights(lr, batchsize); l3->UpdateWeights(lr, batchsize); l4->UpdateWeights(lr, batchsize); Listing 2: Snippet CNN definition with the C++ library. //Layer creation CNN Conv2D*cnv = Conv2D::Create(H, W, C, K_H, K_W, K_C); MaxPooling*pl = MaxPooling::create(out.h, out.w, out.c, 3,3); Layer*l1 = Layer::create(pl1->getLinearFlattenedSize(), 32); Layer*l2 = Layer::create(32, 10); //Forward M3 a = Sigmoid(cnv->convolve2D(input[j])); M3 b = pl->forward(a); M c = Sigmoid(l1->forward(b.flatten())); M d = Softmax(l2->forward(c)); //Backpropagation M d2 = M::MatMul(CrossEntropyPrime(y[j], c), SoftmaxPrime(c)); M d1 = l1->backward(d2) *SigmoidPrime(a); M d0 = l0->backward(d1); M3 dpl = pl1->backward(d0) *SigmoidPrime(aa); cnv1->backward_conv(input[j], dpl); //Update weights: l2->dw += l2->getDelta(d2, a); l2->db += d2; l1->dw += l1->getDelta(d1, b.flatten()); l1->db += d1; //SGD cnv->updateKernels(lr, batch_size); l1->UpdateWeights(lr, batch_size); l2->UpdateWeights(lr, batch_size); E. Training the models on the microcontroller board In order to test the library for on-device training on a microcontroller board, we develop an ML application to support the experimentation on a microcontroller board with the model training provided by the integrated library. We define the neural network models of the CNN and autoencoder model introduced in sections IV-C and section IV-B, respectively, and use the same dimensions. We do the training on the microcontroller with the same datasets that we used for the functional validation of the previous section with Tensorflow/Keras, but train now the models on the microcontroller. Since the dataset does not consist of samples that the microcontroller can acquire by its sensors, and the microcontroller does not have storage hardware, in addition to the code deployed on the microcontroller for training the network, it was also necessary to implement a controller application running on a PC which sent the training samples, one by one, over the serial port connection to the board, for being trained by the neural network there. The controller application was also used to collect the training and inference time. We first verified the compilation of the ML application for different hardware targets. We used the Arduino Portenta H7 core, the Arduino Nano 33 BLE Sense board, and the TTGO T-Beam board with ESP32 microcontroller, for all of which the library compiled successfully. This is because the developed library uses only standard C++ components without any dependencies on frameworks of microcontroller manufacturers or third-party libraries, which sometimes are only ported to specific microcontrollers. The result is relevant since it paths the way for using the library for ML applications in different microcontroller families. Then we flash an ML application with each of the two defined models on the M7 core of the Arduino Portenta H7 (Figure 3). We measure the memory usage for the autoencoder at different points of operation: For the creation, the memory usage was 37088 bytes. In the forward pass memory usage increased to 37844 bytes, including the activations, and in the backward pass, which includes the gradients and the momentum, the memory usage further expands to 63748 bytes. We measure the training and inference time of both neural network models on the microcontroller board. Training the autoencoder with one sample of the credit card dataset took around 1 ms per sample. The inference with the autoencoder model took 0.3 ms. In the CNN experiment, which is trained with the MNIST dataset, the training took 4 ms per sample and the inference around 1 ms. Since in real TinyML applications, the samples are acquired by the sensors connected to the board, the frequency of these sensor values has to be taken into account to determine the actual requirement for the training times of samples. Fig. 3: Board used for on-device training: Arduino Portenta H7 microcontroller board with additional LoRa radio. F. Discussion In the previous section IV-E both networks were chosen with small dimensions which fitted to the Portenta board that has 1MB RAM and 2MB Flash. But RAM, in general, is a critical resource of microcontroller boards. For training larger neural networks on-device, the RAM availability of the microcontroller imposes limitations. If not only inference is done on the board but also on-device training with backprogagation, additional data structures for holding the gradients, and possibly, the momentum, are required. This is different from off-device training, where only memory for the trained model in the forward pass for inference is needed. Therefore, concerning memory consumption of the application, TinyML applications that apply off-device training can have larger neural network models. The library has shown the capacity for defining the building blocks of neural networks for on-device training, but the internal functions are not yet optimized. Recent research contributions such as quantization for on-device training could be integrated in the next steps to reduce memory consumption [13]. V. CONCLUSIONS This paper presented the motivations for on-device training of neural networks on microcontrollers and the opportunity for a library of neural network architectures in C++ to facilitate experimentation. We developed the library and validated it with two implemented models, the autoencoder and CNN, by comparing their performance in a training and validation data set with a reference implementation in Tensorflow/Keras. We showed that the library integrated into an ML application can be successfully flashed to the Arduino Portenta H7 microcontroller board and the two models can be trained on-device. In our next steps, we will also look at extending the training of our models to federated learning. Another line of work is to deeper analyze the resource usage in on-device training and understand how the library implementation can exploit the optimization opportunities. ACKNOWLEDGMENT This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957258 — ASSIST-IoT, and was partially supported by the Spanish Government under contract PID2019106774RB-C21, and the Generalitat de Catalunya as Consolidated Research Group 2021-SGR-01059. REFERENCES [1] P. P. Ray, “A review on tinyml: State-of-the-art and prospects,” Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 4, pp. 1595–1623, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1319157821003335 [2] S. Hymel, C. Banbury, D. Situnayake, A. Elium, C. Ward, M. Kelcey, M. Baaijens, M. Majchrzycki, J. Plunkett, D. Tischler, A. Grande, L. Moreau, D. Maslov, A. Beavis, J. Jongboom, and V. J. Reddi, “Edge impulse: An mlops platform for tiny machine learning,” 2022. [3] V. Rajapakse, I. Karunanayake, and N. Ahmed, “Intelligence at the extreme edge: A survey on reformable tinyml,” ACM Comput. Surv., feb 2023, just Accepted. [Online]. Available: https: //doi.org/10.1145/3583683 [4] J. Choi, Z. Wang, S. Venkataramani, P. I.-J. Chuang, V. Srinivasan, and K. Gopalakrishnan, “Pact: Parameterized clipping activation for quantized neural networks,” 2018. [5] D. Vasconcelos, M. S. Yin, F. Wetjen, A. Herbst, T. Ziemer, A. F¨ orster, T. Barkowsky, N. Nunes, and P. Haddawy, “Counting mosquitoes in the wild: An internet of things approach,” in Proceedings of the Conference on Information Technology for Social Good, ser. GoodIT ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 43–48. [6] S. Disabato and M. Roveri, “Incremental on-device tiny machine learning,” in Proceedings of the 2nd International Workshop on Challenges in Artificial Intelligence and Machine Learning for Internet of Things, ser. AIChallengeIoT ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 7–13. [Online]. Available: https://doi.org/10.1145/3417313.3429378 [7] F. De Vita, G. Nocera, D. Bruneo, V. Tomaselli, and M. Falchetto, “Ondevice training of deep learning models on edge microcontrollers,” in 2022 IEEE International Conferences on Internet of Things (iThings) and IEEE Green Computing & Communications (GreenCom) and IEEE Cyber, Physical & Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics (Cybermatics), 2022, pp. 62–69. [8] H. Ren, D. Anicic, and T. A. Runkler, “Tinyol: Tinyml with onlinelearning on microcontrollers,” in 2021 International Joint Conference on Neural Networks (IJCNN), 2021, pp. 1–8. [9] K. Kopparapu, E. Lin, J. G. Breslin, and B. Sudharsan, “Tinyfedtl: Federated transfer learning on ubiquitous tiny iot devices,” in 2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops), 2022, pp. 79–81. [10] N. Llisterri Gim´ enez, M. Monfort Grau, R. Pueyo Centelles, and F. Freitag, “On-device training of machine learning models on microcontrollers with federated learning,” Electronics, vol. 11, no. 4, 2022. [Online]. Available: https://www.mdpi.com/2079-9292/11/4/573 [11] “Kann: a c library for artificial neural network,” https://github.com/ attractivechaos/kann, accessed: 2023-07-11. [12] “tiny-dnn,” https://github.com/tiny-dnn/tiny-dnn, accessed: 2023-07-11. [13] J. Lin, L. Zhu, W.-M. Chen, W.-C. Wang, C. Gan, and S. Han, “On-device training under 256kb memory,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 22 941–22 954. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2022/file/90c56c77c6df45fc8e556a096b7a2b2e-Paper-Conference.pdf