Exploring the Capabilities of Deep Learning Models for Transport and Human Activity Recognition
Abstract
This thesis explores the transformative potential of deep learning smartphone-based transportation mode detection systems in enhancing urban planning in the city of Barcelona. The core of this thesis is the development and comparison of algorithms, coupled with extensive data analysis and preprocessing techniques, aimed at reliable transport mode detection. We delve into creating a real-time system capable of predicting transport usage patterns. For that, a dataset of smartphone sensor data has been created with examples of journeys using multiple modes of transportation in the metropolitan area of Barcelona. In the deep learning model, we have experimented with architectures combining convolutional networks and LSTMs to finally create a hierarchical model that combines the use of CNNs for feature extraction, with the ability to process time series from the LSTM layers using skip connections. For robust and battery-efficient detection, we have combined this model with statistical techniques, which allow us to detect at an early stage whether the user is moving, standing or walking. This allows not to make excessive use of the deep learning model, which can be costly in mobile devices. Following, an android application is presented which implements the mentioned techniques and presents a simple way to collect mobility data, which can be useful for future studies on urban mobility in the city. Finally, the various ethical, social and environmental issues that such systems may have are studied, describing the privacy and interpretability factors that this tools must comply with.
Full text
id182802 EXPLORING THE CAPABILITIES OF DEEP LEARNING MODELS FOR TRANSPORT AND HUMAN ACTIVITY RECOGNITION GERARD CARAVACA IBÁÑEZ Thesis supervisor: MÓNICAAGUILARIGARTUA Tutor:JAVIERBÉJARALONSO(DepartmentofComputerScience) Degree:Master'sDegreeinArtificialIntelligence Master's thesis School of Engineering Universitat Rovira i Virgili (URV) Faculty of Mathematics Universitat de Barcelona (UB) Barcelona School of Informatics (FIB) Universitat Politècnica de Catalunya (UPC) - BarcelonaTech 23/01/2024
i Abstract This thesis explores the transformative potential of deep learning smartphone-based transportation mode detection systems in enhancing urban planning in the city of Barcelona. The core of this thesis is the development and comparison of algorithms, coupled with extensive data analysis and preprocessing techniques, aimed at reliable transport mode detection. We delve into creating a real-time system capable of predicting transport usage patterns. For that, a dataset of smartphone sensor data has been created with examples of journeys using multiple modes of transportation in the metropolitan area of Barcelona. In the deep learning model, we have experimented with architectures combining convolutional networks and LSTMs to finally create a hierarchical model that combines the use of CNNs for feature extraction, with the ability to process time series from the LSTM layers using skip connections. For robust and battery-efficient detection, we have combined this model with statistical techniques, which allow us to detect at an early stage whether the user is moving, standing or walking. This allows not to make excessive use of the deep learning model, which can be costly in mobile devices. Following, an android application is presented which implements the mentioned techniques and presents a simple way to collect mobility data, which can be useful for future studies on urban mobility in the city. Finally, the various ethical, social and environmental issues that such systems may have are studied, describing the privacy and interpretability factors that this tools must comply with. Index terms - Activity recognition, deep learning, mobile sensors, transportation mode detection, urban mobility, MobilitApp.
ii Acknowledgements I would like to express my sincere gratitude to all those who have supported and guided me throughout the journey of this thesis. First and foremost, my heartfelt thanks to my supervisor, Professor Mónica Aguilar, for giving me the opportunity to work on this project and for guiding me during these months. Additionally, I am profoundly thankful to the members of the SISCOM group and all the study volunteers. Their willingness to share their time and data was indispensable. This research simply would not have been achievable without their invaluable contributions. I would also like to thank Professor Javier Béjar for his valuable feedback. His expertise greatly influenced my research. My gratitude extends to the faculty and staff at the Facultat de Informàtica de Barcelona (FIB). Their support and the opportunities provided have been vital in enhancing my research experience, both in the bachelor’s and master’s degrees. The environment here has been both challenging and inspiring. On a personal note, I extend my deepest gratitude to my family for their relentless love and support. They have been a constant source of inspiration and strength throughout this journey. Additionally, I am immensely thankful to my friends for their incredible support and encouragement. Finally, I acknowledge Autoritat del Transport Metropolità (ATM) for their support and feedback, which was essential for completing this project
iii Contents Abstract i Acknowledgements ii List of Figures v List of Tables vii Introduction 1 1 Preliminary 2 1.1 Context....................................... 2 1.2 Motivation and research objectives . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Methodologyoverview.............................. 3 2 Fundamental knowledge 5 2.1 Urban mobility in Barcelona . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.1 Future of Mobility in Barcelona . . . . . . . . . . . . . . . . . . . . . 7 2.1.2 Our role in urban mobility of Barcelona . . . . . . . . . . . . . . . . 7 2.2 Transport mode recognition task . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1 Applications of transport mode recognition task . . . . . . . . . . . 8 2.2.2 Challenges of transport mode recognition task . . . . . . . . . . . . 8 2.3 Traditional machine learning techniques . . . . . . . . . . . . . . . . . . . . 10 2.3.1 Support Vector Machines (SVM) . . . . . . . . . . . . . . . . . . . . 10 2.3.2 K-Nearest Neighbors (KNN) . . . . . . . . . . . . . . . . . . . . . . 11 2.3.3 Random Forest (RF) . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.4 Deep learning techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.4.1 Multilayer perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.4.2 Convolutional Neural Networks (CNN) . . . . . . . . . . . . . . . . 14 2.4.3 DilatedCNN ............................... 15 2.4.4 Recurrent Neural Networks (RNN) . . . . . . . . . . . . . . . . . . 15 2.4.5 Long short-term memory (LSTM) . . . . . . . . . . . . . . . . . . . 17 2.4.6 Bidirectional LSTM (Bi-LSTM) . . . . . . . . . . . . . . . . . . . . . 18 2.4.7 Attention mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.4.8 Transformers ............................... 21 2.4.9 Transformers in Transport mode recognition . . . . . . . . . . . . . 22 2.5 Performanceevaluation ............................. 22 2.5.1 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.5.2 ConfusionMatrix............................. 23
Contents iv 2.5.3 LearningCurves ............................. 24 2.5.4 Group k-fold cross-validation . . . . . . . . . . . . . . . . . . . . . . 24 2.6 Data acquisition modalities . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.6.1 Motion modality for transport mode recognition . . . . . . . . . . . 25 Advantages of motion modality for transport mode recognition . . 26 Drawbacks of motion modality for transport mode recognition . . 26 2.6.2 Location modality for transport mode recognition . . . . . . . . . . 28 Advantages of location modality for transport mode recognition . 28 Drawbacks of location modality for transport mode recognition . . 28 2.6.3 Ambient modality for transport mode recognition . . . . . . . . . . 28 Advantages of ambient modality for transport mode recognition . 29 Drawbacks of ambient modality for transport mode recognition . . 29 3 Background 31 3.1 Keycontributions................................. 31 3.1.1 Traditional proposals . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 HemminkietAl. ............................. 31 ManzonietAl............................... 32 NhametAl................................. 32 BrezmesetAl................................ 32 Ravi et Al. and Kwapisz et Al. . . . . . . . . . . . . . . . . . . . . . 32 Rosenberg Randleff et Al. . . . . . . . . . . . . . . . . . . . . . . . . 33 3.1.2 Recent contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 GjoreskietAl................................ 33 MuradetAl................................. 33 OkitaetAl. ................................ 33 SongetAl.................................. 33 JeyakumaretAl. ............................. 34 3.2 Dataavailability.................................. 34 3.2.1 Sussex-Huawei Locomotion Dataset (SHL) . . . . . . . . . . . . . . 34 Advantages of the SHL dataset . . . . . . . . . . . . . . . . . . . . . 36 Limitations of the SHL dataset . . . . . . . . . . . . . . . . . . . . . 36 3.2.2 Transport Mode Detection Dataset (TMD) . . . . . . . . . . . . . . . 37 Advantages of the TMD dataset . . . . . . . . . . . . . . . . . . . . 38 Limitations of the TMD dataset . . . . . . . . . . . . . . . . . . . . . 38 3.2.3 CollectyDataset.............................. 39 Advantages of the Collecty dataset . . . . . . . . . . . . . . . . . . . 40 Limitations of the Collecty dataset . . . . . . . . . . . . . . . . . . . 40 4 Our urban mobility dataset 41 5 Model experimentation and refinement 42 5.1 Frameworkdefinition .............................. 42 5.1.1 Startingpoint ............................... 42 Initialpreprocessing ........................... 43 Initialsetsseparation........................... 43 Initial model architecture . . . . . . . . . . . . . . . . . . . . . . . . 43 5.1.2 Effects of sets separation algorithm . . . . . . . . . . . . . . . . . . . 44
Contents v 5.1.3 Effects of window size and overlapping factor . . . . . . . . . . . . 46 5.1.4 Effects of data augmentation . . . . . . . . . . . . . . . . . . . . . . 47 5.1.5 Effects of smoothing technique . . . . . . . . . . . . . . . . . . . . . 49 5.1.6 Effects of outlier elimination . . . . . . . . . . . . . . . . . . . . . . 51 5.1.7 Final configuration of the baseline model . . . . . . . . . . . . . . . 52 5.2 Traditional machine learning baselines . . . . . . . . . . . . . . . . . . . . . 53 5.3 BiLSTM....................................... 55 5.4 Mixedmodel.................................... 56 5.4.1 Regularization .............................. 58 5.4.2 Convolutional Block Variations . . . . . . . . . . . . . . . . . . . . . 60 5.4.3 Activation Function Tuning . . . . . . . . . . . . . . . . . . . . . . . 61 5.4.4 Recurrent Block Variations . . . . . . . . . . . . . . . . . . . . . . . 62 5.5 Hierarchicalmodel ................................ 64 5.6 Transferlearning ................................. 67 5.7 Resultsdiscussion................................. 70 5.7.1 Final Hierarchical model evaluation . . . . . . . . . . . . . . . . . . 73 6 MobilitApp tool for recognition of transportation modes 75 7 Ethical and environmental concerns 76 7.1 Ethicalconsiderations .............................. 76 7.1.1 Privacy................................... 76 7.1.2 Interpretability .............................. 77 7.1.3 Security .................................. 78 7.2 Environmentalimpact .............................. 78 8 Conclusions and Future work 80
vi List of Figures 1.1 National and international organizations that support the project. . . . . . 3 2.1 Number of users (millions) for each public transport per quarter in Barcelona (20162021). [16] ................................. 6 2.2 Categories of deep learning in sensor based human activity recognition challenges, also applicable to transport mode recognition. [14]....... 9 2.3 Example of a linearly separable problem solved by a SVM. [49] . . . . . . 10 2.4 Visualization of the KNN algorithm applied to the Iris dataset. [49] . . . . 11 2.5 Representation of a multilayer perceptron with two hidden layers. [22] . . 13 2.6 (a) The architecture of the LeNet-5 network. (b) Visualization of features in the LeNet-5 network. Each layer’s feature maps are displayed in a different block. [8] .................................. 14 2.7 Example structure for temporal feature extraction applying 1D convolutional layers. [14] ................................. 15 2.8 (a) 1-dilated convolution; each element has a receptive field of 3×3. (b) 2dilated convolution; each element has a receptive field of 7×7. (c) 4-dilated convolution; each element has a receptive field of 15×15. [64]........ 15 2.9 Diagram of a 3-layer recurrent neural network. [27] ............. 16 2.10 Comparison of a RNN cell (above) and an LSTM cell (below). [55] . . . . . 17 2.11 LSTM cell structure. [55]............................. 18 2.12 LSTM-based architecture example for extracting temporal features from sensor signals. [14] ................................ 18 2.13 Bi-LSTM structure applied to NLP [7]...................... 19 2.14 Scaled Dot-Product Attention mechanism [59]. ................ 20 2.15 Multihead attention block [59]. ......................... 21 2.16 The Transformer - model architecture. [59] .................. 22 2.17 Confusion matrix’s simple example [23]..................... 24 2.18 Bias and variance represented in an error’s learning curve [23]. . . . . . . 24 2.19 Axis directions for the accelerometer of the smartphones. [9]........ 27 2.20 Axis directions for the gyroscope of the smartphones. [9] .......... 27 2.21 Functioning of the smartphone magnetometer. [9] .............. 27 2.22 Example of the use of RSS technology in indoor spaces [39]. ........ 30 3.1 Architecture proposed by Hemminki et Al. [29]. ............... 31 3.2 Cumulative duration in hours of each activity in the SHL dataset. [60] . . 35 3.3 Positioning of the device in the data collection process. [60]......... 36 3.4 Preprocessing steps on the TMD dataset. [13]................. 38 3.5 Collecty dataset distribution. [19]........................ 40
List of Figures vii 5.1 LSTM baseline arquitecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.2 Training and validation users distribution after applying the random split configuration. ................................... 44 5.3 Training and validation user distribution after applying the different users splitconfiguration. ................................ 45 5.4 Original dataset distribution, without data augmentation. . . . . . . . . . 47 5.5 Class distribution in the dataset augmented using two variations of data augmentation techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 5.6 Visualization of the effects of Gaussian Smoothing technique with sigma=1, inaCarsample................................... 50 5.7 Learning curves throughout the different epochs of the model trained using the final configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 5.8 Confusion matrix of the model trained using the final configuration. . . . 53 5.9 Confusion matrix of the presented MLP model. . . . . . . . . . . . . . . . . 55 5.10 Learning curves throughout the different epochs of the BiLSTM model trained using the tuned optimization parameters. . . . . . . . . . . . . . . 56 5.11 Base architecture diagram for the mixed model combining LSTM and CNNs. 57 5.12 Learning curves throughout the different epochs of the mixed model trained using using a dropout rate of 20%. . . . . . . . . . . . . . . . . . . . . . . . 59 5.13 Learning curves throughout the different epochs of the model mixed trained using using a spatial dropout rate of 5%. . . . . . . . . . . . . . . . . . . . . 60 5.14 Confusion matrix of the presented mixed model. . . . . . . . . . . . . . . . 63 5.15 Learning curves throughout the different epochs of the mixed model trained using the final configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . 64 5.16 Base architecture diagram for the hierarchical model. . . . . . . . . . . . . 65 5.17 Learning curves throughout the different epochs of the hierarchical model trained using the final configuration. . . . . . . . . . . . . . . . . . . . . . . 66 5.18 Confusion matrix of the presented hierarchical model. . . . . . . . . . . . . 67 5.19 Distributions of the activities in the SHL Preview dataset. [60] ....... 68 5.20 Learning curves throughout the different epochs of the hierarchical model trained in the SHL preview dataset. . . . . . . . . . . . . . . . . . . . . . . 68 5.21 Confusion matrix of the hierarchical model trained in the SHL preview dataset........................................ 69 5.22 Learning curves throughout the different epochs of the hierarchical model trained in the SHL preview dataset and finetuned. . . . . . . . . . . . . . . 69 5.23 Confusion matrix of the hierarchical model trained in the SHL preview datasetandfinetuned. .............................. 70 5.24 Box plot depicting the F1-scores of the LSTM baseline from subsection 5.1.7, the MLP baseline from section 5.2, the mixed model from subsection 5.4.4 and the hierarchical model from section 5.5. The central line in each box represents the median F1-score, the edges of the boxes indicate the interquartile range, and the whiskers extend to the full range of the data, excludingoutliers. ................................ 72 5.25 Bar chart comparing the performance of the hierarchical model from section 5.5 using only one sensor. (M.avg. means Macro Average and W.avg. meansWeightedAverage)............................ 72 5.26 Confusion matrix of the hierarchical model from section 5.5 on the test set. 74
viii List of Tables 2.1 The 10 Spanish cities with the most traffic jams in 2022. [43] ........ 7 2.2 Average battery consumption of the considered sensors. Information taken from many examples found in the literature and some tests made for diversesmartphones................................. 27 3.1 Summary of User Data in the TMD dataset. [13] ............... 37 3.2 Time durations for various activities in the TMD dataset. [13] ....... 37 3.3 Distribution of data by transport mode per user expressed in hours for the Collecty dataset. [19] ............................... 39 5.1 Number of unseen users by transport in the validation set after applying the different users split configuration. . . . . . . . . . . . . . . . . . . . . . 45 5.2 Average performance of the group 5-fold with each of the two separation algorithms. (M.avg. means Macro Average and W.avg. means Weighted Average) ...................................... 45 5.3 Average performance of the group 5-fold with each of the configurations. (M.avg. means Macro Average and W.avg. means Weighted Average) (WS means Window Size and OF means Overlapping Factor. . . . . . . . . . . 46 5.4 Average performance with each of the data augmentation versions. (M.avg. means Macro Average and W.avg. means Weighted Average) . . . . . . . 48 5.5 Classification Reports using Original configuration (left) vs data augmentation base configuration (right). . . . . . . . . . . . . . . . . . . . . . . . . 49 5.6 Average performance using different Gaussian smoothing parameters. (M.avg. means Macro Average and W.avg. means Weighted Average) (The first entry in the table shows the results without smoothing) . . . . . . . . . . . . 50 5.7 Outlier Analysis by User ID and Label . . . . . . . . . . . . . . . . . . . . . 51 5.8 Classification Reports showing average performance of LSTM model trained afteroutliersdetection .............................. 52 5.9 Parameter settings for SVM, RF, KNN, and MLP random searches. Bold values indicate that this is the final value chosen for each parameter. . . . 54 5.10 Average performance using traditional machine learning architectures. (M.avg. means Macro Average and W.avg. means Weighted Average) . . . . . . . 54 5.11 Average performance using different sizes for the BiLSTM layers. (M.avg. means Macro Average and W.avg. means Weighted Average) . . . . . . . 56 5.12 Average performance with each of the regularization methods in the mixed model. (M.avg. means Macro Average and W.avg. means Weighted Average) (Dp refers to the dropout rate, SDp refers to the Spatial dropout rate and L2 refers to the L2 regularization parameter) . . . . . . . . . . . . . . . 59
Chapter 1. Preliminary 4 The next step is the experimentation step, in which both traditional machine learning models and deep learning models, such as convolutional neural networks (CNN), recurrent neural networks (RNN) and Transformers, will be compared and adapted to the task of transport activity recognition. Following this, we will present the implementation of the proposed system after the study carried out. Finally, the report will conclude with the discussion on the impact that the implementation of this type of technology can have on today’s society and the future work proposed.
5 Chapter 2 Fundamental knowledge This chapter sets up the theoretical groundwork on which this thesis relies and offers readers the essential theoretical information to comprehend the following chapters. The first part of the chapter delves into the topic of Mobility in the Barcelona metropolitan area, providing a comprehensive overview of the region’s transportation dynamics. Subsequently, it introduces the Transport Mode Recognition task, describing its significance in the context of mobility analysis. In the pursuit of effective recognition, the chapter explores various machine learning techniques, with a special focus on deep learning methods. Additionally, it covers the critical aspect of evaluation, which ensures the reliability and accuracy of the developed models. Furthermore, the chapter explain the diverse Data Acquisition Modalities utilized in collecting the necessary data for this task. 2.1 Urban mobility in Barcelona During a typical workday in 2019, there were 5.7 million trips made in Barcelona, averaging 3.6 trips per person. A slight decline of 0.3% in total trips was observed in 2021 (last official data available) when excluding professionals like taxi drivers. This data was collected from the annual Mobility Survey on Workdays (EMEF), conducted collaboratively by various local and metropolitan transportation authorities [26]. Public transport accounts for approximately 17% of all journeys made in Barcelona. Various modes of public transport serve every district of Barcelona, with buses, commuter rail, and light rail - encompassing both subway and tram - being the primary options [16]. Figure 2.1 illustrates the quarterly passenger count trends for each of these transport modes from 2016 to 2021. As can be seen in the graph, since 2016 the use of the various modes of public transport in Barcelona has remained fairly stable with some fluctuations. However, as expected, during 2020 a decrease in all types of transport has been observed due to the Covid-19 season. During 2021 a rebound in data is observed without reaching pre-Covid levels. The rebounds in 2021 suggest a recovery phase and possibly a return of confidence in using public transportation. Although last year’s data have not yet been published, it is expected that public transport use has returned to the levels of the pre-Covid years.
Chapter 2. Fundamental knowledge 6 FIGURE 2.1: Number of users (millions) for each public transport per quarter in Barcelona (20162021). [16] On the opposite side, in terms of private transport, almost half the trips, specifically 39%, are made using personal cars, in Barcelona [16]. The recent pandemic has engendered a discernible shift in the populace’s transportation preferences, primarily driven by health and safety concerns associated with public transit systems. The aversion to shared spaces, particularly in the context of public transportation, has prompted an augmented reliance on personal vehicles. This emergent trend has intensified traffic congestion within the city. Empirical data suggests a consequential 29% augmentation in travel durations purely attributable to traffic congestion in Barcelona. In fact, Barcelona is now the city in Spain with the greatest traffic problems according to data from TomTom traffic index [58] (see Table 2.1). This substantial increment is unparalleled in the Spanish context; for instance, Madrid, another significant urban conglomerate, witnesses a comparatively lower 23% increase in travel time due to vehicular congestion [43]. This divergence in urban mobility patterns between the two cities underscores the imperative for tailored transportation strategies, particularly in the post-pandemic era.
Chapter 2. Fundamental knowledge 7 TABLE 2.1: The 10 Spanish cities with the most traffic jams in 2022. [43] City Avg. 10 km time (min) Hours in traffic per year Avg. Speed (km/h) Barcelona 18.3 161 29 Madrid 18 159 29 Valencia 16.3 141 33 Gijón 15.6 127 36 Sevilla 14.8 136 34 Vitoria 14.8 122 38 Málaga 14.6 126 36 Zaragoza 14.5 121 36 Granada 14.3 129 36 Palma de Mallorca 14.1 129 36 2.1.1 Future of Mobility in Barcelona The Mobility Plan in Barcelona [17] showcases a forward-thinking approach towards urban sustainability and livability. By reducing private vehicle use and enhancing public transport efficiency, the plan addresses both environmental concerns and modern urban living standards. The expansion of pedestrian streets and bike lanes, along with the promotion of shared vehicle services, promises a more active, healthy lifestyle among residents. Furthermore, the "Superilla" project and Via Laietana’s transformation contribute to creating communal and green spaces, which are vital for social interactions and mental well-being. However, residents might face a transitional phase, adapting to new mobility patterns and potentially altered traffic conditions. The ambitious goal of shifting the majority of commutes to walking, cycling, and public transport by 2024 means a substantial change in daily routines. Yet, if executed effectively, these alterations could lead to a more accessible and environmentally friendly urban landscape, enhancing the overall living experience in Barcelona. It should be noted that this plan corresponds to Barcelona City Council’s 2023 bid. It may be subject to change with the new candidature. 2.1.2 Our role in urban mobility of Barcelona As can be seen from the data shown in the previous sections, most of the studies on mobility in Barcelona are based on data obtained from citizen surveys. This is why updated data for the last few years has not yet been obtained. This complicates the analysis of the impact of the agreed mobility plan to improve mobility in the city since data collection by conventional methods is slow and complicated, often requiring significant human resources and financial investment. Relying solely on citizen surveys presents a series of limitations. Firstly, these surveys are subject to biases, as people might not accurately recall or honestly report their transportation habits. Secondly, the periodic nature of these surveys means that real-time or frequent data updates are virtually non-existent, making it challenging to monitor rapid changes in mobility patterns or to assess the immediate impacts of newly implemented policies.
Chapter 2. Fundamental knowledge 8 Transitioning to a transport mode detection system using smartphone motion sensors offers a compelling alternative. Such a system ensures near real-time, objective data collection, eliminating biases inherent in personal recall or reporting. Additionally, this digital approach is cost-effective, scalable, and adaptable to rapidly changing scenarios. Leveraging this technology-driven method will greatly enhance Barcelona’s ability to effectively analyze and manage urban mobility, ensuring more informed decision-making in its pursuit of improved transportation dynamics. 2.2 Transport mode recognition task Transport mode recognition is a sub-field of the human activity recognition task that involves identifying and categorizing the mode of transportation used by an individual or an object. This process has various applications and, still today, suppose a number of challenges. 2.2.1 Applications of transport mode recognition task In the realm of urban planning and traffic management, transportation mode recognition provides crucial data to optimize city infrastructure and improve traffic flow. By understanding how people move about the city, authorities can make informed decisions about public transit expansion, bike lane development, and road maintenance. At this point, the task of transport recognition plays a decisive role. This is because it allows the collection of data for the subsequent analysis of urban mobility on a large scale. In the field of healthcare, this technology can contribute to tracking and encouraging physical activity, helping individuals lead healthier lives. Smart wearables and health apps can use transportation mode recognition to monitor users’ activity levels and offer personalized fitness recommendations. Moreover, transportation mode recognition has a profound impact on environmental sustainability. By promoting eco-friendly modes of transport like cycling or walking and discouraging the excessive use of personal cars, we can collectively reduce carbon emissions and alleviate the burden on our planet. 2.2.2 Challenges of transport mode recognition task As summarized in 3.1, much research has been carried out in transport mode recognition. However, this field still faces many technical challenges. Some of the difficulties are shared by other pattern recognition domains, such as computer vision and natural language processing, while others are specific to sensor-based activity detection and need specialized algorithms for real-world applications. In the following, we present a set of categories outlining the challenges. A visual representation of this taxonomy can be found in Figure 2.2. •Data acquisition: Training and evaluating deep learning models require large annotated data. In this context, it is particularly expensive and time-consuming to collect sensory activity data (see section 2.6). •Feature extraction: This challenge is usually shared with other classification problems. For sensor-based transport recognition, it is even a more difficult task because
Chapter 2. Fundamental knowledge 9 there is inter-activity similarity. This means that different transports may have similar characteristics (e.g., walking and running). Therefore, it is difficult to produce distinguishable features to represent activities uniquely. •Data distribution: In this task, the dataset may be unbalanced for three reasons. The first one is class imbalance. This is an important challenge because it is difficult to find large amounts of data on less common transports, such as e-scooter. Apart from that, some transport patterns are user-dependent, which means that different users may have diverse activity styles. Finally, the position or different configuration of the sensors may influence the simulated data. •Computational cost: This task is intended to be used on portable devices such as a smartphone. This type of device has limited computational resources. For this reason, lightweight and easily optimizable models must be generated for this type of device. •Concurrent transports: Ideally, when performing classification tasks, it is taken into account that each sample belongs to only one possible class. However, in this case in today’s public transport it is common for people to walk or take other types of transport with them. This should be treated differently depending on the final application of the system. •Privacy: As the recognition system could potentially record users’ lives continuously, there are risks of personal information disclosure, which make the privacy issue determinant to be analyzed before deploying the system. •Interpretability: Sensory data cannot be read like images or sentences. Furthermore, due to the inherent flaws in sensors, sensory data invariably contains a lot of noise information. Therefore, trustworthy recognition solutions must be able to analyze sensory input and know which aspects of data help with recognition. FIGURE 2.2: Categories of deep learning in sensor based human activity recognition challenges, also applicable to transport mode recognition. [14]
Chapter 2. Fundamental knowledge 10 2.3 Traditional machine learning techniques In the context of transport mode detection using smartphone motion sensors, while deep learning models offer advanced capabilities and intricate pattern recognition, traditional machine learning techniques remain essential to the foundation of predictive analytics. It is for this reason that this type of algorithm will be used as baseline for further experiments. This means that the results of these algorithms should be used to ensure that the accuracy of the complex models is not lower than that achieved with the simple models. This section delves into three of the most pertinent traditional techniques for our context: RandomForest (RF), K-Nearest Neighbors (KNN), and Support Vector Machines (SVM). 2.3.1 Support Vector Machines (SVM) SVM [28] is a non-parametric supervised learning algorithm that operates by finding the hyperplane that best divides the data into classes (in classification tasks), ensuring that the margin between the classes is maximized. For data that is not linearly separable, SVM utilizes the kernel trick, mapping the data into higher dimensions where a separating hyperplane can be found. Figure 2.3 shows the decision function for a linearly separable problem, with three samples on the margin boundaries, called “support vectors”. The hyperparameters used to tune this algorithm are: •C (regularization parameter): Controls the trade-off between maximizing the margin and minimizing classification errors. A smaller value of C creates a wider margin but may misclassify more data points. •Kernel type: Specifies the kernel function to be used (e.g., linear, polynomial, Radial basis function). •Gamma: Determines the shape of the decision boundary. A low gamma value will produce a more flexible curve, while a high value will create a more rigid confined shape. •Degree: Degree of the polynomial kernel function. FIGURE 2.3: Example of a linearly separable problem solved by a SVM. [49]
Chapter 2. Fundamental knowledge 11 2.3.2 K-Nearest Neighbors (KNN) K-Nearest Neighbors (KNN) [66] is one of the simplest yet surprisingly effective clusteringbased supervised machine learning algorithms. Its fundamental premise is that data points that are close in feature space have similar output values, or class labels. For classification tasks, KNN simply stores instances of the training data and works by determining the ’k’ training samples closest in distance to a new point and returning the most common output value among them by majority vote [66]. Figure 2.4 shows a visualization of this clustering algorithm. The main hyperparameters used to tune this algorithm are: •Number of Neighbors (k): The number of neighbors to consider when making classifications. •Distance Metric: The method of calculating distance between data points, e.g. Euclidean, Manhattan, Minkowski, etc. •Weighting: Decides if all neighbors have equal vote or if closer neighbors have a stronger influence on the prediction. FIGURE 2.4: Visualization of the KNN algorithm applied to the Iris dataset. [49] 2.3.3 Random Forest (RF) Random Forest (RF) [12] is an ensemble learning method that builds upon the foundational Decision Tree algorithm. It creates a "forest" of decision trees during training, each constructed using a random subset of the training data and a random subset of the features. When making a prediction, each tree in the forest casts a vote, and the Random Forest aggregates these votes to determine the final output. This ensemble approach both diversifies and stabilizes the Decision Tree model’s predictions. The main hyperparameters used to tune this algorithm are:
Chapter 2. Fundamental knowledge 12 •Number of Trees: Specifies how many decision trees should be built in the forest. •Max Features: The maximum number of features to consider when looking for the best split. •Max Depth: The maximum depth of the tree. •Min Samples Split: The minimum number of samples required to split an internal node. •Min Samples Leaf: The minimum number of samples required to be at a leaf node. •Bootstrap: Whether bootstrap samples are used when building trees. 2.4 Deep learning techniques At its core, sensor based transport mode recognition involves information extraction of sensor’s data embedded in smartphones. These sensors, such as GPS, accelerometers, gyroscopes and magnetometers, collect vast amounts of data about human movements and surroundings. By definition, deep learning is a subset of machine learning, which is basically a neural network with three or more layers. These neural networks attempt to emulate the behaviour of the human brain-though far from matching its capabilities, but allow it to "learn" from large amounts of data. Although a neural network with a single layer can already make approximate predictions, additional hidden layers help to optimize and refine accuracy [31]. One of the remarkable aspects of deep learning in this context is its ability to detect patterns and features within sensor data that would be nearly impossible for humans to detect with other traditional methods. This is why, utilizing a neural network to extract temporal features becomes advantageous when building an end-to-end deep learning model [37]. This end-to-end learning approach streamlines the training process and improve mutual enhancement between feature learning and recognition processes. Numerous deep learning techniques have been employed for the extraction of temporal information, the most common ones for the task of this thesis will be explained below. 2.4.1 Multilayer perceptron The multilayer perceptron, as described in reference [22], forms the foundation of feedforward networks. It consists of a system of simple interconnected neurons, or nodes, as illustrated in Figure 2.5, which is a model representing a nonlinear mapping between an input vector and an output vector. The nodes are connected by weights and output signals which are a function of the sum of the inputs to the node modified by a simple nonlinear transfer, or activation, function. It is the superposition of many simple nonlinear transfer functions that enables the multilayer perceptron to approximate extremely non-linear functions.
Chapter 2. Fundamental knowledge 13 FIGURE 2.5: Representation of a multilayer perceptron with two hidden layers. [22] The multilayer perceptron operates through a structured sequence of layers, each with a specific role in information processing: •Input Layer: This is the initial layer of the network, composed of artificial input neurons. These neurons hold the original data representing external inputs or features. •Hidden Layers: Positioned between the input and output layers, the hidden layers play a pivotal role. They apply transformations to the input data using activation functions and then pass these transformed values to the output layer. Within these hidden layers, the neural network defines its weights, which signify the strength of connections between individual nodes. Weight updates, a critical component, represent the learning phase during neural network training. Adjusting weights helps the network fine-tune its ability to recognize patterns and make accurate predictions. •Output Layer: This final layer of the network provides the algorithm’s output, transmitting the results of the computations performed by the preceding layers. Finally, the learning of the neural network is due to the backpropagation algorithm [62]. This algorithm involves two main steps: the forward pass, where input data is processed through the network to produce an output, and the backward pass, where the error between the predicted output and the actual target is propagated backward through the layers. During the backward pass, weights are updated to minimize this error by moving in the direction that reduces it, with the learning rate controlling the step size. This process is repeated for multiple iterations, gradually improving the network’s ability to make accurate predictions on training data.
Chapter 2. Fundamental knowledge 20 value is computed by a compatibility function of the query with the corresponding key [59]. The most commonly used attention function is Scaled Dot-Product Attention (see Figure 2.14). In this variant, the attention function is computed on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V. The matrix output is as follows: Attention(Q,K,V) = so f tmax(QKT √dk )V(2.1) FIGURE 2.14: Scaled Dot-Product Attention mechanism [59]. The Scaled Dot-Product Attention mechanism, while effective, has its limitations when dealing with complex data structures. To address this, the concept of Multihead Attention was introduced [59]. The primary idea behind Multihead Attention is to allow the model to focus on different parts of the input simultaneously, capturing various types of relationships and dependencies. Instead of using a single set of attention weights, the multihead attention block employs multiple sets, often referred to as "heads". Each head computes its own attention weights and produces its own output vector. These outputs are then concatenated and linearly transformed to produce the final output of the multihead attention block. This process is represented by the following equation: MultiHead(Q,K,V) = Concat(head1, . . . , headn)WO(2.2) Figure 2.15 represents the multihead attention block, taking has the number of heads employed. As can be observed in the figure, after computing the attention scores and obtaining the weighted sum of the Value vectors, the results (for multiple heads in multihead attention) are concatenated. Finally, the output is typically passed through another linear layer to produce the final output.
Chapter 2. Fundamental knowledge 21 FIGURE 2.15: Multihead attention block [59]. 2.4.8 Transformers Transformers [59], employ an attention mechanism that systematically assesses an input sequence’s elements, assigning varying degrees of importance to each element at every step. This novel approach has revolutionized sequential data analysis, progressively surpassing LSTM, which was the previous state-of-the-art model in this domain. To achieve this, the Transformers follow a very characteristic architecture, shown in Figure 2.16. The left part of the architecture corresponds to the Encoder block. The encoder is composed of a stack of N_x identical layers. Every layer consists of two sub-layers. The initial one is a multi-head self-attention mechanism, while the subsequent one is a straightforward, position-wise fully connected feed-forward network. A residual connection around both sub-layers, complemented by layer normalization, is applied. On the other hand, the right part of the architecture corresponds to the Decoder block. The decoder is also composed of N_x identical layers. Besides the two sub-layers in every encoder layer, the decoder incorporates a third sub-layer that executes multi-head attention on the encoder stack’s output. Changes were made to the self-attention section in the decoder to prevent it from looking ahead. By shifting the output by one position, it’s ensured that predictions at any given position are based solely on outputs from preceding positions. Finally, two important components of the transformer architecture are embeddings and positional encodings. Embeddings are a way to convert discrete variables, like words or tokens, into continuous vector representations. Positional Encoding is added to the embeddings before the data is fed into the encoder or decoder. This ensures that the transformer has both the semantic information from the embeddings and the positional information from the positional encoding. This solves an inherent limitation of the standard transformer model, since originally it did not take into account the order of the input. This is because it does not inherently process the data in sequence, like recurrent neural networks (RNNs).
Chapter 2. Fundamental knowledge 22 FIGURE 2.16: The Transformer - model architecture. [59] 2.4.9 Transformers in Transport mode recognition Transformers, with their proficiency in handling sequential data, can be pivotal in transport mode detection using smartphone motion sensors. Their self-attention mechanism allows them to discern specific motion patterns that can characterize different transport modes, like distinguishing the rhythmic movement of walking from the smoother motion of vehicular travel. Moreover, the ability of transformers to integrate data from multiple sensors, such as accelerometers and gyroscopes, further improves their efficacy in this application. However, these advantages come with certain trade-offs. For one, the computational demands of Transformer models are high, potentially posing challenges for real-time processing on resource-constrained smartphones. This complexity also translates to increased power consumption, which is a concern for battery-dependent devices. Additionally, while they excel in rich data environments, transformers can be prone to overfitting when trained on limited datasets. This means that despite their impressive performance during training, they might not generalize well to real-world, unseen data. Lastly, these models require substantial labeled data for effective training, and acquiring such data for every possible transport mode in varying conditions can be a huge challenge. 2.5 Performance evaluation The effectiveness of a classifier is heavily influenced by the attributes of the data it needs to classify. To evaluate its performance in various contexts, a combination of metrics and
Chapter 2. Fundamental knowledge 23 visual methods are employed. Here is a detailed explanation of the evaluation metrics used in this work. 2.5.1 Performance Metrics Performance metrics are an integral component of every machine learning pipeline, serving as vital indicators of progress and success. These metrics provide quantitative measures that help in assessing the effectiveness of a model, offering clear, numerical insights into its performance. To this end, we delve into four key metrics: precision, recall, F1score, and accuracy. Each of these metrics provides unique insights into the performance of the classifier. •Precision: This metric indicates the proportion of correctly predicted positive observations to the total predicted positives. Precision =True Positives (TP) True Positives (TP) +False Positives (FP) (2.3) •Recall: Recall calculates the proportion of actual positives that were correctly identified. Recall =True Positives (TP) True Positives (TP) +False Negatives (FN) (2.4) •F1-Score: The F1-Score is the weighted average of precision and recall. F1-Score =2×Precision ×Recall Precision +Recall (2.5) •Accuracy: It measures the proportion of true results (both true positives and true negatives) among the total number of cases examined. Furthermore, while these metrics are intuitive, they may not always be the best metric for imbalanced classes, which is common in transport mode detection. Two variants of the above metrics are used in this case for a more informative performance analysis: •Macro Average: This averages the metric independently for each class and then takes the average (hence treating all classes equally). •Weighted Average: This accounts for class imbalance by weighting the average of the metric in favour of the most abundant class. It is calculated for each class label, and the average is weighted by the number of true instances for each label. 2.5.2 Confusion Matrix Aconfusion matrix is a vital tool in the evaluation of classifiers. It offers a visual representation of a classifier’s performance. In this matrix, each row represents the instances in an actual class, and each column corresponds to the instances in a predicted class (see Figure 2.17). The matrix aids in understanding the types of errors made by the classifier, for example, mistaking walking for cycling or driving for public transit.
Chapter 2. Fundamental knowledge 24 FIGURE 2.17: Confusion matrix’s simple example [23]. 2.5.3 Learning Curves Training curves are graphical representations that show the evolution of the model’s performance over time during the training process. These curves typically include loss curve and accuracy curve. Monitoring these curves helps in detecting issues such as overfitting or underfitting, bias and variance allowing for necessary adjustments to the model’s training process (see Figure 2.18). FIGURE 2.18: Bias and variance represented in an error’s learning curve [23]. 2.5.4 Group k-fold cross-validation Group k-Fold Cross-Validation was utilized as the primary model evaluation technique in this study, tailored for datasets with distinct groupings. This method is especially appropriate for data segregated by entities such as users, which is the case, ensuring no overlap between training and testing sets. The dataset is divided into ’k’ distinct groups with different users on each group, with each one successively used as a test set while the others form the training set. This strategy guarantees comprehensive utilization of data for both training and validation purposes and ensures that each group is entirely
Chapter 2. Fundamental knowledge 25 excluded from the training data during its turn as the validation set, thus bolstering the evaluation’s robustness and relevance. Following the Group k-Fold Cross-Validation, an Mann-Whitney U test [45] was conducted to statistically compare models. This statistical test, also known as the Wilcoxon rank-sum test, is a non-parametric statistical test used to compare two independent samples to determine whether there is a difference in their distribution. It is particularly useful for comparing models in cases where the data does not follow a normal distribution, which is a common assumption for many parametric tests. By applying Mann-Whitney U test to the results obtained from each fold of the cross-validation, the study could determine if there is a significant difference in the performance of the two models. 2.6 Data acquisition modalities In this section, we will explore various methods and technologies for gathering data related to human activity and movement. The section will delve into three primary modalities: motion, location, and ambient sensing. For each modality, the advantages and disadvantages will be examined, offering an objective overview of their strengths and limitations in data acquisition for diverse purposes. 2.6.1 Motion modality for transport mode recognition The performance of an activity recognition system depends crucially on the sensor modality used. There are diverse types of sensors, such as wearable sensors, ambient sensors, and location sensors. However, in this work, we are going to focus on the most commonly used sensors available in smartphones, wearable sensors [20]. •Accelerometers are devices used to measure acceleration, specifically the rate of change in an object’s velocity. They are typically measured in metres per second squared (m/s2) or G-forces and operate at sampling frequencies ranging from tens to hundreds of Hz. They provide a tri-variate time series due to their three axes (see Figure 2.19). •Gyroscopes measure orientation and angular velocity, with the unit of angular velocity being radiants per second (rad/s). Like accelerometers, they also operate at sampling rates ranging from tens to hundreds of Hz. Gyroscopes are often integrated with accelerometers, and also provide three axes of data (see Figure 2.20). •Magnetometers, on the other hand, are commonly used wearable sensors and are usually combined with accelerometers and gyroscopes into an inertial unit. They measure changes in the magnetic field at a specific location, using Tesla (T) as the measurement unit and having sampling rates in the tens to hundreds of Hz. Magnetometers, like the others, also have three axes. They can be used to estimate the three-dimensional orientation of the device relative to the Earth’s magnetic north. Figure 2.21 depicts the functioning of a magnetometer.
Chapter 2. Fundamental knowledge 26 Advantages of motion modality for transport mode recognition Using a combination of accelerometer, gyroscope, and magnetometer sensors is advantageous for activity recognition because it provides a comprehensive and robust dataset. This combination offers the following benefits: •Comprehensive Data: A wide variety of information is provided by these three sensors when taken together, including linear acceleration (from the accelerometer), angular velocity (from the gyroscope), and orientation with regard to the Earth’s magnetic field (from the magnetometer). This extensive dataset records many facets of motion and orientation, enabling a more complete comprehension of the user’s motions. •Orientation Awareness: The magnetometer offers important details regarding the user’s orientation with respect to the Earth’s magnetic field. This information is particularly useful for classifying modes like walking and bicycling, where changes in direction play a big role. •Real-Time Capability: The data from these sensors can be processed in real-time, allowing for instantaneous mode recognition. •Privacy Considerations: Unlike GPS, which can be highly intrusive in terms of user privacy, accelerometer, gyroscope, and magnetometer data can be processed without revealing the user’s exact location. •Widespread availability: All modern smartphones come equipped with these sensors as standard hardware components. This availability makes it exceptionally convenient and cost-effective to implement transport mode recognition on a large scale, as users do not need to invest in additional hardware or devices. •Battery consumption: The battery consumption of the smartphone sensors is much lower than that caused by other sources of information. This can be seen in detail in Table 2.2. Drawbacks of motion modality for transport mode recognition On the other hand, this method of data collection also has some drawbacks, which are explained below: •Data Noise: Sensor data can be noisy, especially in real-world scenarios. Vibrations, shocks, and external interference can introduce errors into the data. This noise can affect the accuracy of transport mode recognition algorithms, leading to incorrect results. •Dependency on Device Placement: The placement of sensors within a device can affect their performance. Different smartphones and wearables may have sensors located in slightly different positions, leading to variations in data collection and recognition accuracy. •Integration Challenges: Integrating sensor-based recognition into apps or devices can be technically challenging and may require specialized knowledge in signal processing, machine learning, and software development.
Chapter 2. Fundamental knowledge 27 •Calibration and Sensor Drift: Over time, sensor values can drift due to temperature changes or wear and tear. Maintaining accurate and calibrated sensors can be a challenge. FIGURE 2.19: Axis directions for the accelerometer of the smartphones. [9] FIGURE 2.20: Axis directions for the gyroscope of the smartphones. [9] FIGURE 2.21: Functioning of the smartphone magnetometer. [9] TABLE 2.2: Average battery consumption of the considered sensors. Information taken from many examples found in the literature and some tests made for diverse smartphones. GPS (update 15sec) WiFi (update 15sec) ACC MAG GYR 250 mA 125 mA 0.23 mA 6.8 mA 6.1 mA
Chapter 2. Fundamental knowledge 28 2.6.2 Location modality for transport mode recognition Location-based information can also be used to identify a user’s mode of transportation or activity. This method uses information from the Global Positioning System (GPS) or other location services to obtain a person’s current position and rate of movement. The following are some benefits and drawbacks of utilizing location modality for identifying transport modes: Advantages of location modality for transport mode recognition •High-Level Information: Location data can provide high-level information about a user’s activity, such as whether they are indoors, outdoors, in a car, on foot, or using public transportation. This can be valuable for recognizing transport modes. •Easy Integration: GPS libraries are usually integrated into smartphone frameworks. This makes it easier to work with this type of data. •Contextual information: location data provides contextual information about the trip. Drawbacks of location modality for transport mode recognition •Indoor Limitations: Location data can be less accurate indoors or in areas with poor GPS signal reception. Recognizing transport modes or activities indoors can be challenging. •Lack of Fine-Grained Information: Location data may not provide fine-grained information about specific transport modes, such as distinguishing between different types of vehicles (e.g., car, bus, train). •Battery Consumption: Continuous GPS usage can consume a significant amount of power, potentially affecting device battery life. •Privacy Concerns: Gathering location data raises privacy concerns, as it can reveal a user’s whereabouts. App developers must handle location data responsibly and transparently to address user privacy concerns. •Dependency on Location Services: Transport mode recognition based on location data relies on the availability and accuracy of location services on the user’s device. Any issues with these services can affect the reliability of the recognition system. 2.6.3 Ambient modality for transport mode recognition Ambient sensors, such as Wi-Fi, RFID (Radio-Frequency Identification), and radar, can be used for transport mode recognition as well. •Wi-Fi is a local-area wireless network connection technology that uses a transmitter to send signals to a receiver. The basis of WiFi-based human activity recognition is that human’s movements and locations interfere with the signals’ propagation path from the transmitter to the receiver, including both the direct propagation path and the reflecting propagation path.
Chapter 2. Fundamental knowledge 29 •RFID uses electromagnetic fields to automatically identify and track the tags attached to objects, which contain electronically stored information. RSS is the most widely adopted tool for RFID-based activity recognition; an example is shown in Figure 2.22. The working mechanism is that human’s movements would change the single strength received by the RFID reader [61]. •Radars. Unlike WiFi and RFID whose transmitters and receivers are placed on opposite sides, radar transmitters and antennas are mounted on the same side of users. The Doppler effect is the basis of the radar-based system [39]. Advantages of ambient modality for transport mode recognition •Multi-occupant detection: Each person’s presence and movements can influence the Wi-Fi signals differently, making it possible to recognize and distinguish multiple occupants in the monitored area. This capability is useful for applications like occupancy sensing in smart homes or tracking the number of people in a public space. •Indoor localizing: Wi-Fi can work well indoors, where GPS signals may be weak or unavailable. •Network-based data: Wi-Fi can provide information about the availability of Wi-Fi networks, which can be used as a context clue for transport mode recognition. Drawbacks of ambient modality for transport mode recognition •Infrastructure deployment: Deploying an RFID or Radar infrastructure can be costly and time-consuming, making it less practical for wide-scale use. •Battery consumption: Depending on the radar system’s design, it may consume a significant amount of power, which can be a drawback for battery-powered devices. In addition, having the phone’s Wi-Fi activated also consumes a significant amount of energy. •Limited Outdoor Accuracy: Ambient based recognition may be less accurate for outdoor activities, where GPS is more reliable.
Chapter 3. Background 36 FIGURE 3.3: Positioning of the device in the data collection process. [60] As with any dataset, the SHL presents a unique set of advantages and disadvantages that researchers and analysts should consider. Below is an examination of its strengths and potential limitations. Advantages of the SHL dataset •Sensor variability: With data points ranging from basic motion sensors like accelerometer, gyroscope, and magnetometer to ambient light and audio, the dataset provides a multi-dimensional view of the surroundings and actions. •Integrated Third-Party Data: The integration of Google’s activity recognition API output gives a benchmark to compare with any custom activity recognition models. •Battery and Connectivity Data: Information about battery level and temperature, WiFi and mobile network details provide insights into device status, which can be crucial for real-world, continuous monitoring applications. •Positioning information: The dataset gives information about the exact positioning of the device at each moment of the data collection. This can help to detect if the generated models fail at any particular position. Limitations of the SHL dataset •Overwhelming Complexity: The sheer breadth and granularity of data might be overkill for simple applications. Handling such vast data requires more processing power and advanced algorithms. •Potential Redundancies: Some data types might overlap in the information they provide. For instance, accelerometer data combined with gyroscope and orientation might lead to redundant information in some scenarios. •Hardware Dependency: the dataset only comes from a specific device, which can make the accuracy of the developed models very dependent on the device in use. •Limited Diversity in Participants: The dataset comes from only 3 members, which significantly limits the diversity of the data. This might not represent a broader population, affecting the generalizability of any models trained on this dataset.
Chapter 3. Background 37 3.2.2 Transport Mode Detection Dataset (TMD) The TMD dataset [13] was assembled by researchers at the University of Bologna who gathered sensor data from thirteen volunteer subjects, comprising ten males and three females (see Table 3.1). The primary aim of this dataset is to classify a range of activities, which include walking, driving a car, standing still, being on a train, and riding a bus. Altogether, the dataset consists of 226 labeled files. These files represent over 31 hours of data with breakdowns as follows: 26% of the data is designated as walking, 25% as driving a car, 24% as standing still, 20% as being on a train, and 5% as being on a bus (see Table 3.2). TABLE 3.1: Summary of User Data in the TMD dataset. [13] ID Sex Age Occupation Device Android Version U1 M 30 student LG G2 5.0.2 U2 F 27 student Sony XPERIA Z3 Compact D5803 6.0.1 U3 M 30 student Nexus 5 7.0 U4 M 36 office worker Huawei Honor 5X 6.0.1 U5 M 36 stage director Huawei P8 Lite 6.0.1 U6 M 27 researcher Samsung galaxy s3 neo 4.4.2 U7 M 32 cameramen Samsung S7 6.0.1 U8 F 32 bartender Huawei Tag-l01 5.1 U9 F 24 student Motorola Moto G 5.1 U10 M 22 student Huawei P9 7.0 U11 F 31 office worker Nexus 5 7.0 U12 M 31 researcher Samsung Galaxy S6 6.0.1 U13 M 60 retired Nexus 5 7.0 TABLE 3.2: Time durations for various activities in the TMD dataset. [13] Bus Car Still Train Walking Total 01:44:35 07:53:50 07:29:35 06:20:25 08:20:25 31:48:50 In the initial data preprocessing phase, the researchers undertook a series of data cleaning operations. These included the removal of measurements from non-pertinent sensors and ensuring the positivity of values from the sound and speed sensors, among other adjustments. It is noteworthy that some sensors, especially the ambient ones like sound, light, and pressure, as well as the proximity sensors, produced a single data value. This data was directly incorporated into the dataset. On the other hand, other sensors yielded multiple values because they were associated with a coordinate system, implying that their outputs were heavily influenced by orientation. For most of these, the team adopted an orientation-independent metric, termed magnitude. Following the data cleaning process, the dataset was segmented into time windows, each of either 5 seconds or half a second’s duration. Subsequent to this division, four distinct features (min, max, dev.std, and mean) were extracted from every sensor. Figure 3.4 serves as a summary of the preprocessing steps.
Chapter 3. Background 38 FIGURE 3.4: Preprocessing steps on the TMD dataset. [13] Advantages of the TMD dataset •Diverse Data Collection: The dataset includes sensor data from thirteen volunteer subjects, enhancing its representativeness. This diversity accounts for ten males and three females, providing a balanced gender distribution. •Thorough Preprocessing: An initial data cleaning phase ensures the removal of unnecessary measures, the correction of sensor values, and the adaptation of certain sensor outputs. This results in a polished and easily usable dataset. •Accurate User Information: the dataset explanatory paper provides precise information on the users and the devices used to collect the data. This can help to see if the models produced are able to generalize to different smartphone models. Limitations of the TMD dataset •Fixed Time Window: The data is segmented into specific time windows (either 5 seconds or half a second). This fixed windowing might not capture all nuances of certain activities or might oversimplify others. •Restricted Number of Features: The dataset confines its scope to only four features extracted from each sensor. This limitation might omit potentially valuable information or insights that other features could provide. •Constrained Data Collection Duration: While over 31 hours of data seems extensive, in the context of capturing diverse human activities and behaviors, this duration might be considered limited. It might not encompass the full spectrum of variabilities and patterns inherent to each activity. •Limited number of classes: It would have been interesting if the dataset had included other transportation such as: run, scooter, bike, subway, etc.
Chapter 3. Background 39 3.2.3 Collecty Dataset The Collecty dataset [19] offers a unique perspective into the transportation habits within Croatia, specifically in the City of Zagreb. The data has been accumulated using the mobile application "Collecty" on Android devices. Key sensors, including the accelerometer, gyroscope, and magnetometer, have been instrumental in data collection. Ensuring data privacy, the format remains raw and anonymized. The dataset boasts contributions from 15 participants across various age groups, spanning a data collection period of 5 months. These participants were tasked to activate the mobile application, which then recorded sensor data according to their mode of transportation. To ensure accuracy, upon reaching their destinations, participants validated their routes via the app’s displayed digital map. Table 3.3 shows the trips made by each user present in the database. TABLE 3.3: Distribution of data by transport mode per user expressed in hours for the Collecty dataset. [19] User ID Walk Run Bike Car Bus Train Tram E-scooter Total 17 7.92 0.00 0.00 6.24 0.00 2.84 0.00 0.14 17.13 23 10.76 0.06 0.18 0.54 22.73 0.35 5.05 0.00 39.68 24 0.29 0.00 0.00 1.78 0.00 0.00 0.00 0.00 2.06 25 0.71 0.00 0.00 9.47 0.00 0.00 0.75 0.00 10.92 29 29.88 0.00 0.00 4.60 0.92 39.32 0.00 0.18 74.9 37 0.00 0.00 0.00 28.13 0.00 0.00 0.00 0.00 28.13 39 1.27 0.00 0.00 1.47 0.82 3.78 0.00 0.00 7.35 40 6.22 0.02 0.00 0.27 5.66 0.00 2.27 0.00 14.43 18 6.59 3.31 9.41 15.8 0.00 0.00 0.00 5.98 41.09 20 0.02 0.00 0.00 0.00 0.00 0.00 0.14 0.00 0.15 26 0.03 0.00 0.00 0.12 0.00 0.00 0.00 0.00 0.14 27 1.87 0.00 0.00 0.83 0.00 0.78 0.00 0.00 3.47 28 0.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.04 31 0.16 0.00 0.00 0.29 0.39 0.00 0.00 0.00 0.85 35 0.09 0.00 0.00 2.07 0.00 0.00 0.00 0.00 2.16 Total 65.85 3.39 9.59 71.61 30.52 47.07 8.21 6.3 242.54 This dataset provide a a wide range of transports described in Figure 3.5. Tramway sees the least amount of use, tallying just a bit over 10 hours. In contrast, Walking and Car are the predominant methods of transport, with both nearing 70 hours. Bus travel is also a notable mode, accounting for approximately 30 hours. While Bike and E-scooter have diminished numbers, with Bike hours slightly surpassing 10 and E-scooter just falling short of that mark. Train travel, on the other hand, is quite prominent, registering a little more than 50 hours. The least frequent activity is Running, which is nearly absent on the representation.
Chapter 3. Background 40 FIGURE 3.5: Collecty dataset distribution. [19] Advantages of the Collecty dataset •Wide Range of Transports: The dataset covers various modes of transportation, including tramways, walking, cars, buses, bikes, e-scooters, trains, and running. This offers a holistic view of transportation habits. •Detailed Duration Data: The dataset provides exact hours of usage for each user and mode of transport, allowing for quantitative analysis and comparisons. •Anonymized Data: Ensures privacy and confidentiality of participants while still providing valuable insights. Limitations of the Collecty dataset •Geographical Limitation: The data pertains only to transportation networks within Croatia, specifically the City of Zagreb, which could limit its generalizability. •Application Not Public: The Collecty mobile application used for data collection is not publicly available, which could hinder replication or validation of the study by other researchers. •No Information on Devices: The dataset does not provide information regarding the specific devices used for data collection, which could impact the accuracy and consistency of the sensor data. In conclusion, the current landscape lacks a comprehensive public dataset capable of effectively training machine learning models for predicting transportation modes in urban settings. An ideal dataset for this purpose would encompass data from a broad spectrum of urban residents, encompassing variou transportation methods. Additionally, it is crucial to ensure meticulous control over data generation processes and other relevant parameters. Within the framework of the MobilitApp [2] and Mobilytics [3] projects, conducted in partnership with the ATM [1], there is a concerted effort to create and provide such a robust, public dataset dedicated to urban mobility data, serving both research and urban planning objectives.
41 Chapter 4 Our urban mobility dataset This chapter is dedicated to describing the methodology behind the creation and refinement of the dataset that is central to this study. It starts with the deatiled exposition of the sensors employed. Then the data collection process is detailed. Subsequently, the preprocessing techniques are described. Furthermore, some data augmentation techniques are explained, with the intention of being applied to improve the dataset. Then Feature Extraction follows, detailing the transformation of preprocessed data into a structured feature set, including the Outlier Detection process. Finally, the chapter culminates with Resultant Dataset section, where the characteristics of the final dataset —ready for exploratory and confirmatory data analysis— are thoroughly described, setting the stage for the research covered by this project. The subsequent content of this chapter is governed by a confidentiality agreement.
42 Chapter 5 Model experimentation and refinement The Experimentation chapter serves as a detailed investigative work with the aim of exploring and analyze the performance of various computational models in the field. It begins with the definition of the framework, documenting the experimentation with different framework configurations, including variations in prepocessing. Once the basis of the experiments has been established, the next step in this chapter is to evaluate the performance of conventional machine learning algorithms, which is used as a baseline for further experiments. Then, different deep learning architectures are explored in conjunction with the attempted finetuning of a pretrained mdoel, trying to push the boundaries of classifying performance. Finally, the Results Discussion section synthesizes the experimental findings, providing a comprehensive overview of the outcomes and main conclusions of the experimentation. 5.1 Framework definition In the context of this work, the framework refers to the set of methods, techniques, and algorithms used to carry out the research experiments. Therefore, in this section we will explain the framework taken as a template and present some experiments on variations of the preprocessing steps to determine the best configuration for the following experiments. 5.1.1 Starting point This project does not start from scratch; in this case, we will start from a baseline developed by researchers and previous students of the SISCOM group [6][33]. This section will describe this basis, which will serve as a foundation for further experiments. The main things to comment on about the framework used so far are the configuration of the sliding window, the overlapping factor, the separation between training and validation sets, the architecture of the model used, and some other preprocessing steps.
Chapter 5. Model experimentation and refinement 43 Initial preprocessing The preprocessing steps carried out in the pre-project phase were the same as those described in ??. However, it is important to emphasize the critical role of specific parameters in this process, particularly the size of the sliding window and the overlapping factor, both of which significantly influence the system’s performance. The baseline configuration for these parameters is detailed as follows: •Window size: The window size is established at 200 timesteps. This dimension dictates the volume of data the model processes at any given time, influencing the system’s ability to interpret and learn from temporal patterns within the data. •Overlapping factor: An overlapping factor of 50% has been selected. This indicates that each data window shares half of its content with the succeeding window. Initial sets separation In any project involving machine learning models, it is imperative to first segregate the dataset into training, validation, and test sets before proceeding to model evaluation. This segmentation is fundamental to accurately evaluate the model’s performance in realworld applications and to conduct a thorough analysis of potential overfitting. For the baseline, the dataset was divided primarily through random partitioning. This approach was executed to ensure that the distribution of classes and users remained consistent across all three datasets - training, validation, and testing. Initial model architecture FIGURE 5.1: LSTM baseline arquitecture. The architecture previously used up to the time of this project was an LSTM model. The particularities of these architectures are explained in subsection 2.4.5. In this case, the particular architecture consists of two LSTM (Long Short-Term Memory) layers interspersed with dropout layers, followed by a dense layer at the end. The LSTM layers, vary in their complexity and output dimensions, with the first layer preserving the sequence’s length, using 200 units, and the second condensing it into a single vector, using 512 units. Dropout layers are employed to prevent overfitting, using a 20% dropout rate, randomly deactivating a fraction of the neurons during training [54]. The final dense layer serves as the output layer, applying the Softmax activation. The diagram in Figure 5.1 shows the global architecture of the model. This architecture will be trained using the following parameters for the baseline experiments: •Adam optimizer
Chapter 5. Model experimentation and refinement 44 •Learning rate: 0.001 •Batch size: 64 •Num. epochs: 70 •Loss function: Categorical crossentropy •Random seed: 42 The evaluation of each experiment was carried out by running the experiment in a 5group fold and obtaining the average performance. The experiments were developed in a PC equipped with: at the core a 13th Generation Intel Core i9-13900F processor, complemented by 64 GB of physical memory, the graphics processing is managed by an Nvidia Geforce RTX 3060 Lite with 12GB of video memory and a 512 GB solid-state drive. 5.1.2 Effects of sets separation algorithm In this first experiment, the effects of two types of dataset separation algorithms will be studied. The objective is to compare how the performance of the model varies using the classical separation algorithm, explained in section 5.1.1, and using a slightly more complex one. The standard algorithm allocates data into training, validation, and test sets randomly, ensuring the distribution of classes and users remains consistent. The newer algorithm, however, places emphasis on populating the validation and test sets with users who are not present in the training set. This strategy aims to better simulate real-world scenarios where the model encounters users it has not previously seen. FIGURE 5.2: Training and validation users distribution after applying the random split configuration.
Chapter 5. Model experimentation and refinement 45 FIGURE 5.3: Training and validation user distribution after applying the different users split configuration. In the comparative analysis between Figure 5.2 and Figure 5.3, a uniform distribution of users is observed across the training and validation sets in the first configuration, while the second configuration exhibits a distinct user distribution in the validation set. Furthermore, Table 5.1 elucidates the number of novel users by transportation mode. Despite efforts in dataset partitioning to achieve diversity, a perfectly balanced representation across all classes remains challenging due to the variability in user numbers per category. Nonetheless, this dataset splitting strategy more closely mirrors real-world conditions, potentially enhancing the robustness and generalizability of machine learning models. TABLE 5.1: Number of unseen users by transport in the validation set after applying the different users split configuration. Transport Unseen users Bike 4 Bus 3 Car 3 Subway 2 Motorbike 1 Run 1 Stationary 7 Train 3 Tram 1 Walk 10 e-Scooter 2 For a comprehensive comparison, the configurations and model parameters outlined in previous sections are consistently applied to both algorithms. Consequently, Table 5.2 presents the average performance metrics derived from a 5-fold cross-validation for each configuration. Initially, the random separation approach may appear superior due to its significantly higher performance metrics. However, this enhanced performance is attributed to the model’s overfitting to data from the same users. In contrast, when the model trained with this approach is evaluated on unseen data, there is a notable decrease in its f1-score, which drops to 74%. Therefore, for this study, the more suitable algorithm is identified as the one used in the second configuration, known as the different users separation algorithm, due to its similarity to the real environment. TABLE 5.2: Average performance of the group 5-fold with each of the two separation algorithms. (M.avg. means Macro Average and W.avg. means Weighted Average) Split conf. F1-score M.avg. P M.avg. R M.avg. F1 W.avg. P W.avg. R W.avg. F1 random 93 91 89 90 93 93 93 dif. users 79 71 67 67 79 79 78
Chapter 5. Model experimentation and refinement 52 TABLE 5.8: Classification Reports showing average performance of LSTM model trained after outliers detection Class Precision Recall F1-score Support Bike 0.44 0.46 0.45 195 Bus 0.80 0.79 0.80 1957 Car 0.85 0.92 0.88 1805 Motorbike 0.47 0.45 0.46 171 Run 0.98 0.92 0.95 357 Stationary 0.75 0.67 0.71 495 Subway 0.78 0.81 0.80 1781 Train 0.44 0.18 0.25 349 Tram 0.84 0.86 0.85 1152 Walk 0.87 0.91 0.89 1265 e-Scooter 0.66 0.68 0.67 276 Accuracy 0.80 9803 Macro avg 0.72 0.70 0.70 9803 Weighted avg 0.79 0.80 0.80 9803 5.1.7 Final configuration of the baseline model As a summary of the previous sections, different configurations of the preprocessing steps have been tested. As a result, it has been decided to establish the following configuration as the basis of the project: •Separation algorithm: different users separation. •Windows size: 512 •Overlap factor: 50% •Data augmentation: DAbase •Smoothing techniques: no smoothing •Outliers detection technique: Mahalanobis Distance FIGURE 5.7: Learning curves throughout the different epochs of the model trained using the final configuration.
Chapter 5. Model experimentation and refinement 53 In this first experiment, the implemented configuration yielded an F-score of approximately 80% and a macro average F1-score of 70% . Detailed insights into these results are elaborated upon in preceding sections. Nonetheless, there exists substantial scope for improvement. The confusion matrix, as illustrated in Figure 5.8, reveals certain misclassifications, notably between transport modes like train and subway. An improvement strategy may involve alterations to the model architecture. For instance, employing residual networks could facilitate the extraction of features across varied scales. Additionally, alternative architectures such as Convolutional Neural Networks (CNNs) might prove more efficient in feature extraction. A further branch for improvement lies in model training. As depicted in Figure 5.7, the model exhibits early signs of overfitting. This suggests that the optimization parameters currently in use are suboptimal, indicating another potential area for refinement. The subsequent experiments will delve into these aspects, using the parameters established in the current study as foundational benchmarks. FIGURE 5.8: Confusion matrix of the model trained using the final configuration. 5.2 Traditional machine learning baselines Once the experiment setup is set, the first study to be carried out is to test the effectiveness of machine learning models trained with the manually extracted features. For this, the process described in detail in ?? will be carried out. With the configuration of the dataset set in the previous section it has been obtained that the PCA manages to reduce the number of features from 99 to 38 making the number of features feasible for training using traditional architectures. On this basis, four different models have been tested: RF, SVM,KNN and MLP. For all of them, different combinations of hyper-parameters have been tested using Random Search [10], in order to obtain the maximum effectiveness.
Chapter 5. Model experimentation and refinement 54 All tested combinations are shown in Table 5.9. In addition, the parameters that result in the best performance for each model are also highlighted. TABLE 5.9: Parameter settings for SVM, RF, KNN, and MLP random searches. Bold values indicate that this is the final value chosen for each parameter. Model Parameter Values SVM C 0.1, 1, 10 kernel rbf, poly degree 2, 4 gamma scale, auto, 0.1, 1 RF n_estimators 50, 100, 200, 500 max_features auto, sqrt, log2 max_depth None, 15, 30, 50 min_samples_split 2, 5, 10 min_samples_leaf 1, 2, 3, 4 bootstrap True, False KNN n_neighbors 1, 2, 3, 4, 5...31 weights uniform, distance metric euclidean, manhattan, minkowski MLP n_neurons [128, 64], [256, 128], [512, 256], [512, 512] learning_rate 0.001, 0.01, 0.1 The data presented in Table 5.10 clearly demonstrates the efficacy of manually extracted features in transport prediction for this dataset. Notably, the computational resources required for training these models are significantly lower compared to the LSTM approach, making it a viable alternative for systems constrained by limited resources. However, as anticipated, the peak F1-score achieved with these models, at 76%, does not match the 80% attained by the previously discussed LSTM model. Despite this, it remains a valuable approach to assess the efforts invested in prior experiments. Moreover, these results serve as a useful baseline for comparative analysis. For additional information on the accuracy of the model in the different classes, the confusion matrix is shown in Figure 5.9. TABLE 5.10: Average performance using traditional machine learning architectures. (M.avg. means Macro Average and W.avg. means Weighted Average) Model F1-score M.avg. P M.avg. R M.avg. F1 W.avg. P W.avg. R W.avg. F1 SVM 75 62 61 60 73 75 74 RF 76 64 62 61 75 76 75 KNN 74 61 60 60 73 74 73 MLP 76 64 65 64 77 76 76
Chapter 5. Model experimentation and refinement 55 FIGURE 5.9: Confusion matrix of the presented MLP model. 5.3 BiLSTM In this experiment, an enhancement of the LSTM model’s performance is tried through the implementation of a Bidirectional LSTM (BiLSTM) architecture. As detailed in subsection 2.4.6, the BiLSTM, unlike its LSTM counterpart, processes data sequences bidirectionally, potentially offering a more comprehensive analysis of temporal data patterns. This research involves a meticulous examination of various training parameters and architectural modifications to optimize the model’s efficacy. The exploration of training methodologies includes the evaluation of different optimization algorithms, namely RMSprop and SGD [18]. Additionally, the investigation extends to assessing the impact of varying learning rate values on the model’s training stability. Further, the study incorporates the weight decay approach and an adaptive momentum technique, though these modifications initially led to training destabilization. A notable breakthrough was achieved with the adaptation of the loss function, incorporating class-weighting for less-represented classes in the training set. This adjustment aims to enhance the model’s sensitivity to these classes. Moreover, a smoothing parameter is integrated into the loss function to mitigate overfitting. The best results were obtained with the following configuration: •Optimizer: Adam •Learning rate: 0.001 •Batch size: 64 •Num. epochs: 70 •Loss function: Weighted Categorical Crossentropy
Chapter 5. Model experimentation and refinement 56 •Smoothing parameter: 0.1 Figure 5.10 shows the training curves throughout the different epoafter training the BiLSTM model with the specified parameters. As can be seen, compared to the curve shown in Figure 5.7, the new loss function is better adapted to the training data and therefore the training is better guided by it, showing a more stable training. FIGURE 5.10: Learning curves throughout the different epochs of the BiLSTM model trained using the tuned optimization parameters. On the model side, the experiment replicates the structure of the initial LSTM model but substitutes LSTM layers with BiLSTM layers. Furthermore, the study delves into determining the optimal size for the BiLSTM layers, tailored to the specifics of the dataset in use, thus seeking to maximize the model’s performance. For this purpose, three sizing configurations for the BiLSTM layers have been tested: [128, 128], [128, 256], [256, 256]. The results of the experiments with the different sizes are shown in Table 5.11. Despite the efforts exerted in testing various hyper-parameter configurations, the best results of the experiments performed with this architecture do not surpass the previous best results. However, they are relatively similar in the case of the second configuration by using a model with 700k less parameters. TABLE 5.11: Average performance using different sizes for the BiLSTM layers. (M.avg. means Macro Average and W.avg. means Weighted Average) Size Params. F1-score M.avg. P M.avg. R M.avg. F1 W.avg. P W.avg. R W.avg. F1 [128, 128]0.5M 78 67 70 68 79 78 78 [128, 256] 1.1M 79 69 70 69 79 79 79 [256, 256]2.1M 78 68 69 68 77 78 77 5.4 Mixed model This section explores a mixed architecture model, combining convolutional and recurrent layers, following the initial testing of models solely based on recurrent networks. For this purpose, we builded a base model inspired by the architecture proposed by Tang et Al. in [57]. Consequently, several experiments with variations of this architecture and its hyper-parameters will be performed.
Chapter 5. Model experimentation and refinement 57 The designed architecture is shown in Figure 5.11. The architecture depicted takes input from the three sensors. Each sensor feed passes through a series of convolutional blocks (Conv Block), which are likely composed of convolutional layers with batch normalization, dropout, and max pooling as indicated in the highlighted key for one Conv Block. The Nparameter corresponds to the number of filters of the convolution and the K parameter corresponds to the kernel size. After feature extraction, the outputs of the convolutional blocks from each sensor channel are concatenated and fed into a LSTM layer. Finally there are a dense layer (also known as a fully connected layer), which processes the features learned by the LSTM, and a dense output layer with a softmax activation function which outputs the probabilities for each transport. FIGURE 5.11: Base architecture diagram for the mixed model combining LSTM and CNNs. The primary goal of this mixed architecture is to synergize the feature extraction capabilities of convolutional networks with the sequence processing strengths of recurrent networks. The training parameters selected for this study are based on those used in the
Chapter 5. Model experimentation and refinement 58 previous section, with modifications to optimize training for deeper models. Specifically, the learning rate has been reduced, and the number of epochs has been increased. The training parameters for the experiments in this section are outlined below: •Optimizer: Adam •Learning Rate: 1×10−4 •Batch Size: 64 •Number of Epochs: 100 •Loss Function: Weighted Categorical Crossentropy •Smoothing Parameter: 0.1 To develop the most effective classification model for this dataset, we conducted several experiments in the following sequence: 1. Regularization: Testing various configurations to prevent overfitting, including different Dropout rates, learning rates, and other regularization techniques. 2. Convolutional Block Variations: Experimenting with different layer configurations in the convolutional block. 3. Activation Function Tuning: Evaluating various activation functions, including ReLU, Leaky ReLU, and GeLU. 4. Recurrent Block Variations: Assessing different approaches in the recurrent layer, such as LSTM, BiLSTM, and a lightweight model without a recurrent layer. 5.4.1 Regularization Regularization in Deep Learning is known as a technique used to prevent overfitting. Regularization works by adding additional information or constraints to the model to simplify it, making it less likely to capture the noise in the training data. The two techniques covered by this experiment are Dropout and L2 regularization. Dropout [54] randomly skip neurons during training, which helps in preventing over-reliance on any one node and encourages a distributed representation of features. A variation of the classic dropout technique is the Spatial dropout [38]. It addresses the structure aspects of CNNs by dropping out entire feature maps from the convolutional layers instead of individual neurons. This forces the network to maintain robustness not just at the neuron level but also at a higher level of abstraction On the other hand, L2 regularization [15] adds the squared value of the weights to the loss function. It encourages the model weights to be small, but not necessarily zero. The experimental framework of this study involves varying dropout ratios and assessing the potential synergy of combining both regularization techniques. The conducted experiments include: • Classical Dropout at dropout rates of 20% and 10%. • Spatial Dropout at dropout rates of 10% and 5%. • Combination of Spatial Dropout and L2 regularization.
Chapter 5. Model experimentation and refinement 59 Results, as detailed in Table 5.12, suggest that both Spatial Dropout and L2 regularization contribute to improved model generalization. Notably, dropout rates exceeding 20% not allows the model to learn fine grain data features. An analysis of the learning curves further elucidates these findings. The comparison of the learning curves reveals that a 20% dropout rate leads to stagnant loss oscillations, impeding learning. In contrast, lower spatial dropout rates exhibit more dynamic training progress and enhanced learning capabilities. TABLE 5.12: Average performance with each of the regularization methods in the mixed model. (M.avg. means Macro Average and W.avg. means Weighted Average) (Dp refers to the dropout rate, SDp refers to the Spatial dropout rate and L2 refers to the L2 regularization parameter) Configuration F1-score M.avg. P M.avg. R M.avg. F1 W.avg. P W.avg. R W.avg. F1 Dp: 0 74 66 67 67 74 74 74 Dp: 0.2 70 60 63 60 71 70 69 Dp: 0.1 74 64 66 65 75 74 74 SDp: 0.1 75 66 66 66 75 75 75 SDp: 0.05 75 67 68 67 76 75 75 SDp: 0.05, L2: 0.001 76 69 69 68 76 76 76 A detailed assessment of the regularization techniques’ effectiveness is facilitated through an examination of the learning curves. Comparative analysis of Figure 5.12 and Figure 5.13 reveals distinct outcomes for different dropout rates. Specifically, at a 20% dropout rate (Figure 5.12), the loss exhibits persistent oscillation within a narrow range, indicative of the model’s inability to effectively learn from the training data. Conversely, Figure 5.13, representing a scenario with a reduced spatial dropout rate, demonstrates a more dynamic training process. Despite some instability, this lower dropout rate is associated with enhanced learning outcomes. The fluctuation in loss, although present, does not hinder the model’s learning capacity to the same extent as observed in the higher dropout rate scenario. With all this in mind, the configuration chosen to continue with the following experiments is 5% spatial dropout, combined with L2 regularization in the fully connected layer. FIGURE 5.12: Learning curves throughout the different epochs of the mixed model trained using using a dropout rate of 20%.
Chapter 5. Model experimentation and refinement 60 FIGURE 5.13: Learning curves throughout the different epochs of the model mixed trained using using a spatial dropout rate of 5%. 5.4.2 Convolutional Block Variations The convolutional block is the most important part of the model for feature extraction. The achievement of these blocks should allow the model to extract the features from the data that allow the rest of the model to differentiate between the various modes of transport. This section details the exploration of different configurations for convolutional blocks, focusing on layer arrangements and convolution layer parameters. The conducted experiments, aimed at optimizing feature extraction capabilities, are summarized as follows: • A comparison of classical convolutions and dilated convolutions, with an increasing dilation rate correlating to block depth: 2, 4, 8, 8. Details on dilated convolutions are provided in subsection 2.4.3. • Variations in kernel size for convolutions: [10, 7, 5, 5, 5] compared to [7, 5, 3, 3]. • Implementing multiple convolutions within each convolutional block. • Investigating the optimal placement of the batch normalization layer, whether before or after the convolution process. These experiments have been run sequentially, so that for each experiment the parameters of the previous best experiment are retained. As can be seen in the results table (see Table 5.13), the dilated convolutions are a great improvement in the capabilities of the model to extract features from a large stream of temporal data. In addition, the increased dilation ratio helps the model to extract features at different levels in each block of convolutions. In contrast, it has been observed that a higher number of convolutions staked in the same block does not help to improve the results. On the other hand, it has been detected that the best position for the batch normalization layer is before the convolution for this particular architecture.
Chapter 5. Model experimentation and refinement 61 TABLE 5.13: Average performance with each of the convolutional block configurations in the mixed model. (M.avg. means Macro Average and W.avg. means Weighted Average) (The difference between the second and the third experiment is that in the third experiment an extra convolution per block is added) Configuration F1-score M.avg. P M.avg. R M.avg. F1 W.avg. P W.avg. R W.avg. F1 dilations [10, 7, 5, 5] 78 67 68 67 78 78 78 dilations [7, 5, 3, 3] 80 70 69 69 80 80 80 dilations [7, 5, 3, 3] x2 80 68 67 67 78 80 79 dilations [7, 5, 3, 3], BN after conv 80 70 68 68 79 80 79 As a result, the best configuration extracted from this part of the study is the one using dilated convolutions with kernel sizes of 7, 5 and 3, and with the initial block’s structure. 5.4.3 Activation Function Tuning The activation function in an artificial neural network node is a mathematical function that determines the output of that node given a set of inputs and their corresponding weights. This function plays a crucial role in the network’s ability to capture and represent complex patterns and relationships in the data. It introduces non-linearity into the model, enabling the network to learn and perform more than just simple linear transformations. This is why this section has been devoted to testing variations of the activation function of the convolution layers and the fully connected layer. In this case it has been decided to experiment with ReLU, Leaky ReLU [63] and GeLU [30] as activation functions. •ReLU (Rectified Linear Unit): Defined as f(x) = max(0, x), ReLU is efficient, setting negative inputs to zero and keeping positive values unchanged. It aids in alleviating the vanishing gradient problem but can cause inactive neurons due to its zero output for negative values. •Leaky ReLU: It modifies ReLU by allowing a small gradient when inactive, defined as f(x) = xfor x>0 and f(x) = αxfor x≤0, where αis a small constant. This prevents neurons from becoming inactive. •GELU (Gaussian Error Linear Unit): A smoother function defined as f(x) = xΦ(x), with Φ(x)being the Gaussian distribution’s cumulative distribution function. GELU allows probabilistic gating of inputs, offering nuanced activation behavior and demonstrating efficacy especially in natural language processing. TABLE 5.14: Average performance with each of the activation functions in the mixed model. (M.avg. means Macro Average and W.avg. means Weighted Average) Act. Fun. F1-score M.avg. P M.avg. R M.avg. F1 W.avg. P W.avg. R W.avg. F1 ReLU 80 70 69 69 80 80 80 Leaky ReLU 79 69 69 68 79 79 79 GELU 78 67 68 67 78 78 78
Chapter 5. Model experimentation and refinement 68 the same Huawei Mate 9 smartphone. Therefore, this would not be a good represetative dataset to be used as a starting dataset for a trandfer learning process followed by a fine tuning hase with the own dataset. This is precisely one of our aims in the MobilitApp [2] project in collaboration with the ATM, we aim to generate a large representative dataset of urban mobility data collected from a huge number of citizens in Barcelona. Thus, our MobilitApp public dataset could be used by other researchers and urban planners as a starting point to develope their particular predicte machine learning based models. Returning to the SHL dataset, the data distribution, depicted in Figure 5.19, reveals that the preview dataset, despite being preliminary, encompasses more hours of data for most transport modes compared to other datasets. However, it covers a smaller range of transport types. FIGURE 5.19: Distributions of the activities in the SHL Preview dataset. [60] The methodology employed involved utilizing the model configuration established in the preceding section, the hierarchical model in Figure 5.16, maintaining identical parameters and training algorithms. For this specific iteration, a training duration of 25 epochs proved sufficient to yield notable results on the SHL preview dataset. The effectiveness of this approach is evident in the learning curves depicted in Figure 5.20 and the confusion matrix presented in Figure 5.21, where the model achieved an F1-score of 92%. This outcome underscores the model’s proficiency in adapting to and performing well on the dataset in question. Note, however, that this dataset is composed of data generated by only 3 users and all of them use the same smartphone model. Therefore, its representativeness and usefulness is limited. FIGURE 5.20: Learning curves throughout the different epochs of the hierarchical model trained in the SHL preview dataset.
Chapter 5. Model experimentation and refinement 69 FIGURE 5.21: Confusion matrix of the hierarchical model trained in the SHL preview dataset. Once the base model was obtained, it was decided to perform finetuning by freezing the convolutional blocks and reducing the learning rate to 0.5e-4. This is a very common practice in finetuning. This approach solidifies the feature extraction mechanism and eases the training process by maintaining certain initial weights of the model. However, no better results than those obtained previously have been achieved, with the F1-score stopped at 76%. This may be due to the fact that the task carried out with the preview dataset is clearly easier to complete than the one proposed in this project, since it has fewer modes of transport, a reduced number of users and a single mobile device. Further details of the results of this experiment are shown in Figure 5.22 and Figure 5.23. FIGURE 5.22: Learning curves throughout the different epochs of the hierarchical model trained in the SHL preview dataset and finetuned.
Chapter 5. Model experimentation and refinement 70 FIGURE 5.23: Confusion matrix of the hierarchical model trained in the SHL preview dataset and finetuned. 5.7 Results discussion This section delves into a comparative analysis of four distinct models, evaluated through a robust methodological framework. The approach adopted for this comparison involved group-fold cross-validation with five folds, executed across three different random seeds. This procedure culminated in a total of 15 runs per model, ensuring a comprehensive assessment of each model’s performance consistency and resilience to varying data splits. In particular, the models compared in this section are the two best models of the previous sections and the two baseline models: the LSTM baseline from subsection 5.1.7, the MLP baseline from section 5.2, the mixed model from subsection 5.4.4 and the hierarchical model from section 5.5. To ascertain statistically significant differences in performance among the models, the Mann-Whitney U test [45] was utilized, as explained in subsection 2.5.4. This nonparametric test is particularly adequate for this analysis, given its suitability for comparing distributions without the need for normality assumptions. Each model’s set of F1-scores across the 15 runs was compared against those of the other models, using the Mann-Whitney U test to determine if any observed differences in performance were statistically significant or merely the result of random variation. To begin with the analysis, as a reminder of the previous sections, Table 5.19 shows a summary of the results obtained with the models mentioned above. However, as already mentioned, no empirical conclusions can be drawn from metrics alone. In contrast, Table 5.20 shows the statistical comparison between the hierarchical model, which seems to be the best according to the metrics, and all the other models. In this context, the p-value
Chapter 5. Model experimentation and refinement 71 corresponds to the probability that the observed results would occur under the null hypothesis, which in this case is the assumption that there is no difference between the two models. To interpret this value, 0.05 is normally used as a threshold. Consequently, a value below this threshold indicates that the observed data is very unlikely under the null hypothesis. This leads to the rejection of the null hypothesis, suggesting that there is a statistically significant difference between the two models. As can be seen in the table all p-values are below this threshold which indicates that the difference in the results of the hierarchical model is significant enough to confirm that this model offers better results overall for these data in this task. TABLE 5.19: Summary of the average performance with each of the LSTM baseline from subsection 5.1.7, the MLP baseline from section 5.2, the mixed model from subsection 5.4.4 and the hierarchical model from section 5.5. Results obtained from previous sections. (M.avg. means Macro Average and W.avg. means Weighted Average) Model Params. F1-score M.avg. P M.avg. R M.avg. F1 W.avg. P W.avg. R W.avg. F1 LSTM baseline 1.8M 80 72 70 70 79 80 80 MLP baseline 0.01M 76 64 65 64 77 76 76 Mixed model 2.4M 80 70 69 69 80 80 80 Hierarchical model 1.3M 82 72 72 72 71 81 82 TABLE 5.20: P-values resulting from comparing the highlighted models with the hierarchical model from section 5.5 using the Mann-Whitney U test [45]. Note that a value below 0.05 indicates a significant difference in the results. Models p-value LSTM baseline - Hierarchical model 3.57e−5 MLP baseline - Hierarchical model 3.39e−6 Mixed model - Hierarchical model 4.2e−3 On the other hand, for a more illustrative view of the comparative models, Figure 5.24 is shown. Each box in the plot delineates the interquartile range (IQR) of the F1-scores, with the median value conspicuously marked by a central horizontal line. The "whiskers" of the plot extend to cover the full spread of the data, providing insight into the variability and reliability of each model. This graphical representation is crucial for understanding the comparative strengths and potential limitations of each model, and in this one it can be seen that for most cases the hierarchical model is superior to the others, which indicates that it is the one that offers the best performance. Finally, to conclude the study, we wanted to show the performance of the hierarchical model applied to each of the sensors separately, as shown in Figure 5.25. It can be seen that each of the sensors separately is not able to obtain good results, which indicates that each of them is useful for transport mode classification since the combination of them results in a much higher performance.
Chapter 5. Model experimentation and refinement 72 FIGURE 5.24: Box plot depicting the F1-scores of the LSTM baseline from subsection 5.1.7, the MLP baseline from section 5.2, the mixed model from subsection 5.4.4 and the hierarchical model from section 5.5. The central line in each box represents the median F1-score, the edges of the boxes indicate the interquartile range, and the whiskers extend to the full range of the data, excluding outliers. FIGURE 5.25: Bar chart comparing the performance of the hierarchical model from section 5.5 using only one sensor. (M.avg. means Macro Average and W.avg. means Weighted Average)
Chapter 5. Model experimentation and refinement 73 5.7.1 Final Hierarchical model evaluation As a reminder from section 5.5, the hierarchical model combines convolutional networks with LSTM in a pyramidal structure that allows the processing of the extracted features at different levels using skip connections. The final configuration of the model is as follows: •LSTM size: 128 •Number of filters in CNN: N = [32, 64, 128, 128, 512, 256] •Kernel size: K=[7,5,3,3] •Dilated convolutions •Activation function: ReLU •Regularization: Spatial dropout of 5% and L2 regularizer of 0.001 •Optimizer: Adam with 1e-4 of learning rate •Loss Function: Weighted Categorical Crossentropy with 0.1 of smoothing •Number of epochs: 100 •Batch size: 64 After all the previous analysis the last thing to do is to evaluate the final model (hierarchical model) in the testing set. The obtained performance is depicted in Table 5.21 and Figure 5.26. As a summary, the difference between the validation and test results is small, indicating that the model has not been overfitted to the validation test. TABLE 5.21: Classification Report of the hierarchical model from section 5.5 on the test set. Class Precision Recall F1-score Support Bike 0.59 0.40 0.48 198 Bus 0.84 0.80 0.82 1866 Car 0.84 0.91 0.87 1818 Motorbike 0.50 0.51 0.50 199 Run 0.98 0.93 0.96 305 Stationary 0.66 0.74 0.70 598 Subway 0.86 0.79 0.82 1835 Train 0.21 0.07 0.11 326 Tram 0.86 0.93 0.90 1194 Walk 0.86 0.94 0.90 1215 e-Scooter 0.57 0.80 0.67 251 Accuracy 0.81 9805 Macro avg 0.71 0.71 0.70 9805 Weighted avg 0.80 0.81 0.80 9805
Chapter 5. Model experimentation and refinement 74 FIGURE 5.26: Confusion matrix of the hierarchical model from section 5.5 on the test set.
75 Chapter 6 MobilitApp tool for recognition of transportation modes While the deep learning model constitutes the cornerstone for predicting transportation modes, its practical application necessitates integration within a comprehensive system. This system encompasses several critical components, including real-time data acquisition, preprocessing, the transport prediction, and feedback to the user. This chapter details the development and implementation of an Android application named MobilitApp [2], embodying an end-to-end system that seamlessly integrates data collection and model evaluation processes. The subsequent content of this chapter is governed by a confidentiality agreement.
76 Chapter 7 Ethical and environmental concerns In transport mode detection systems setting up a balance between technological advancement and ethical, environmental responsibility is a paramount concern. This chapter delves into the multifaceted ethical and environmental considerations surrounding the deployment and operation of these systems, establishing a guideline that ensure fair use, secure data storage, environmental stewardship, and sustainable practices. 7.1 Ethical considerations This section examines the ethical considerations specific to the application of Artificial Intelligence (AI) in transport mode detection systems. Addressing these ethical aspects is crucial to ensure the responsible deployment and acceptance of such technologies in urban environments. The main aspects to take into account when talking about ethics applied to the study of artificial intelligence are: privacy, interpretability and security. 7.1.1 Privacy As seen in the previous chapters, the main applications of transport mode detection systems involve monitoring the activities of humans during their trip. Since the manner of driving, walking, running, or even using a phone on public transport changes between users, it may be possible for an adversary to infer from triangulations between samples which sample belongs to which user. Specifically, in the case of deep learning-based systems, the characteristics of these models pose a risk for the disclosure of sensitive user characteristics. Iwasawa et Al. in [32] investigated the privacy issue of using Convolutional Neural Networks as human activity recognition models. Their studies revealed that, although the CNN was initially trained using a cross-entropy loss focused solely on activity classification, the resulting CNN features unexpectedly demonstrated a significant capacity for user discrimination. Utilizing these CNN features, a straightforward logistic regressor was able to attain an impressive 84.7% accuracy in classifying users. This contrasts markedly with the meager 35.2% accuracy achieved by the same classifier when applied to raw sensor data. Consequently, it becomes crucial to consider and address the potential privacy risks inherent in deep learning models originally designed for this task.
Chapter 7. Ethical and environmental concerns 77 There are multiple branches of study in the field of data privatization. And in this section, we will cover two of them: Transformations and Perturbation. In the case of Transformation. this technique involves using an adversarial loss function during the training process of a model to minimize the accuracy of identifying specific private information. This method was explored by researchers like Iwasawa et al. [32], who integrated adversarial loss with standard activity classification loss to reduce user identification accuracy. However, adversarial loss functions can complicate the end-to-end training process, often leading to unstable convergence. On the other hand, Perturbation is an alternative strategy to data transformation for addressing privacy concerns in activity recognition. This method involves modifying data to balance privacy with recognition accuracy. Lyu et al. [41] introduced two data perturbation mechanisms: Random Projection and Repeated Gompertz, aiming to achieve an optimal balance between maintaining privacy and ensuring accurate recognition. Random Projections reduce data dimensionality by projecting it onto a lower-dimensional space using random matrices, maintaining distance relationships while enhancing privacy. In case of Repeated Gompertz, a mathematical model is applied, typically used for growth processes, to systematically alter data, obscuring specific features while preserving useful statistical properties. 7.1.2 Interpretability Sensory data is unreadable for humans, specially in the case where more than one sensor is used in a time window. Given the differing importance of modalities and time intervals, interpreting neural networks is essential to understand the underlying factors influencing model decisions. For instance, in identifying a user’s activity like walking, it is crucial to determine which specific modality and time interval are key determinants. As a result, enhancing the interpretability of deep learning methods is emerging as a significant trend within the transport mode recognition community. Some progress has been made in this field and there are several ways to represent model knowledge. We are presenting Feature Visualization and Attentive Selection. Feature Visualization in the context of interpretable deep learning involves illustrating how neural networks prioritize and process different parts of input data. The core concept is to automatically discern the importance of each input segment, focusing on salient parts for improved accuracy while disregarding less important elements. Researchers have developed methods to visually represent these features, allowing for an understanding of how certain features correlate with specific activities. Additionally, techniques like those proposed by Nutter et al. [47] involve transforming sensory data into image formats, enabling the application of visualization tools for more straightforward interpretability of the data. On the other hand, Attentive Selection applies neural attention mechanisms to deep learning models, enabling them to concentrate on a subset of inputs considered most relevant. Various studies have utilized the attention mechanism as a tool for interpreting the behaviors of deep models, providing insights into which aspects of the input data the models consider most critical for their decision-making processes [65].
BIBLIOGRAPHY 84 [27] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org. [28] M.A. Hearst, S.T. Dumais, E. Osuna, J. Platt, and B. Scholkopf. Support vector machines. IEEE Intelligent Systems and their Applications, 13(4):18–28, 1998. [29] Samuli Hemminki, Petteri Nurmi, and Sasu Tarkoma. Accelerometer-based transportation mode detection on smartphones. 11 2013. [30] Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415, 2016. [31] IBM. Deep learning: what is and how it works, 2021. Accessed: 2023-10-04. [32] Yusuke Iwasawa, Kotaro Nakayama, Ikuko Yairi, and Yutaka Matsuo. Privacy issues regarding the application of dnns to activity-recognition using wearables and its countermeasures by use of adversarial training. pages 1930–1936, 08 2017. [33] Jaume Planas i Planas. Treball de Final de Màster. Improving the MobilitApp tool using deep learning models to automatically identify citizens activity including sustainable transport modes. https://upcommons.upc.edu/handle/2117/382385, 2021. [34] J. Jeyakumar, Eun Sun Lee, Zhengxu Xia, Sandeep Singh Sandha, Nathan Tausik, and Mani B. Srivastava. Deep convolutional bidirectional lstm based transportation mode recognition. Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, 2018. [35] Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. CoRR, abs/1506.02078, 2015. [36] Jennifer R. Kwapisz, Gary M. Weiss, and Samuel Moore. Activity recognition using cell phone accelerometers. SIGKDD Explor., 12:74–82, 2011. [37] Yann LeCun, Y. Bengio, and Geoffrey Hinton. Deep learning. Nature, 521:436–44, 05 2015. [38] Sanghun Lee and Chulhee Lee. Revisiting spatial dropout for regularizing convolutional neural networks. Multimedia Tools and Applications, 79:1–13, 12 2020. [39] Xinyu Li, Yuan He, and Xiaojun Jing. A survey of deep learning-based human activity recognition in radar. Remote. Sens., 11:1068, 2019. [40] Huichao Liu, Ying Feng, and Liguo Zhang. Transportation mode identification based on smartphone. Proceedings of the World Congress on Intelligent Control and Automation (WCICA), 2015:5349–5354, 03 2015. [41] Lingjuan Lyu, Xuanli He, Yee Wei Law, and Marimuthu Palaniswami. Privacypreserving collaborative deep learning with application to human activity recognition. pages 1219–1228, 11 2017. [42] G. Mclachlan. Mahalanobis distance. Resonance, 4:20–26, 06 1999. [43] ABC Motor. Ciudades españolas con más atascos atascos, 2023. Accessed: 2023-1019.
BIBLIOGRAPHY 85 [44] Abdulmajid Murad and Jae-Young Pyun. Deep recurrent neural networks for human activity recognition. Sensors, 17:2556, 11 2017. [45] Nadim Nachar. The Mann-Whitney U: A Test for Assessing Whether Two Independent Samples Come from the Same Distribution. Tutorials in Quantitative Methods for Psychology, 4, 03 2008. [46] Ben Nham, Kanya Siangliulue, and Serena Yeung. Predicting mode of transport from iphone accelerometer data. 04 2012. [47] Mark Nutter, Catherine H. Crawford, and Jorge Ortiz. Design of novel deep learning models for real-time human activity recognition with mobile phones. 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2018. [48] Tsuyoshi Okita and Sozo Inoue. Recognition of multiple overlapping activities using compositional cnn-lstm model. pages 165–168, 09 2017. [49] Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V. and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P. and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E. Scikit-learn: Machine Learning in Python, 2011. [50] Nishkam Ravi, Nikhil Dandekar, Preetham Mysore, and Michael Littman. Activity recognition from accelerometer data. volume 3, pages 1541–1546, 01 2005. [51] Zabic M Rosenberg Randleff L., Bundgaard Wanscher J. Distributed travel mode estimation. 2012. [52] Sima Siami-Namini, Neda Tavakoli, and Akbar Siami Namin. The performance of lstm and bilstm in forecasting time series. In 2019 IEEE International conference on big data (Big Data), pages 3285–3292. IEEE, 2019. [53] Xuan Song, Hiroshi Kanasugi, and Ryosuke Shibasaki. Deeptransport: Prediction and simulation of human mobility and transportation mode at a citywide level. In International Joint Conference on Artificial Intelligence, 2016. [54] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014. [55] Ralf Staudemeyer and Eric Morris. Understanding lstm – a tutorial into long shortterm memory recurrent neural networks. 09 2019. [56] Qinrui Tang and Hao Cheng. Feature pyramid bilstm: Using smartphone sensors for transportation mode detection, 2023. [57] Qinrui Tang, Kanwal Jahan, and Michael Roth. Deep cnn-bilstm model for transportation mode detection using smartphone accelerometer and magnetometer. 06 2022. [58] TomTom. Tomtom traffic index: Measuring congestion worldwide. https://www. tomtom.com/traffic-index/, 2023. Accessed: 2023-10-19.
BIBLIOGRAPHY 86 [59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [60] Lin Wang, Hristijan Gjoreski, Mathias Ciliberto, Paula Lago, Kazuya Murao, Tsuyoshi Okita, and Daniel Roggen. Three-year review of the 2018–2020 SHL challenge on transportation and locomotion mode recognition from mobile sensors. 6 2020. [61] Yanwen Wang and Yuanqing Zheng. Modeling rfid signal reflection for contactfree activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2:1 – 22, 2018. [62] Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550–1560, 1990. [63] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical Evaluation of Rectified Activations in Convolutional Network, 2015. [64] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions, 2016. [65] Dalin Zhang, Lina Yao, Kaixuan Chen, Sen Wang, Pari Delir Haghighi, and Caley Sullivan. A graph-based hierarchical attention model for movement intention detection from eeg signals. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 27(11):2247–2253, 2019. [66] Min-Ling Zhang and Zhi-Hua Zhou. Ml-knn: A lazy learning approach to multilabel learning. Pattern Recognition, 40(7):2038–2048, 2007. [67] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. CoRR, abs/1911.02685, 2019.