UAV State Estimation and Trajectory Prediction using Transformer-based Neural Networks and Feature-based Visual Odometry

UAV State Estimation and Trajectory Prediction using Transformer-based Neural Networks and Feature-based Visual Odometry Yiannis Grigoriou, Nicolas Souli, Panagiotis Chrysanthou, Panayiotis Kolios, and Georgios Ellinas Abstract— Unmanned aerial vehicles (UAVs) are increasingly relied upon in a variety of critical applications, including infrastructure inspection, search-and-rescue, and traffic monitoring. While modern UAVs are typically equipped with global positioning system (GPS), inertial measurement unit (IMU) modules, and often include safeguards against adverse environmental conditions, they remain susceptible to sensor malfunctions and signal disruptions. These challenges have led to the need for robust, GPS-free solutions capable of maintaining accurate trajectory prediction and state identification. This work proposes a real-time multi-task learning framework for UAVs that employs Transformer-based neural networks to perform simultaneous trajectory prediction and state identification. The proposed system enables GPS-free UAV operations with the employment of a feature-based visual odometry algorithm that is implemented and fused with telemetry data to achieve accurate localization. The proposed system is implemented (hardware and software modules) in a functional prototype and validated through extensive outdoor experiments using a custom-built dataset, demonstrating strong performance and improved prediction accuracy in GPS-denied environments. The results demonstrate the framework’s ability to enhance the autonomy and reliability of UAV systems in challenging operational scenarios. I. INTRODUCTION Unmanned aerial vehicles (UAVs) have become one of the most promising and rapidly advancing technologies in numerous application areas. Their ability to operate autonomously and remotely has improved conventional applications and also introduced innovative solutions in various technological fields such as agriculture, disaster management, search-and-rescue, and infrastructure monitoring [1], [2], [3]. Although technological advances in UAVs are rapidly evolving, they are still highly vulnerable to environmental influences [4], jamming attacks [5], and sensor errors, leading to the need for a robust, precise, and reliable system for state detection and trajectory prediction. Current research efforts have focused on optimizing state identification or trajectory forecasting as separate tasks, using neural network models such as support vector machines (SVMs) [6], long short-term memory (LSTMs) [7], [8], and recurrent neural networks (RNNs) [9]. Also, optical flow Y. Grigoriou, N. Souli, P. Chrysanthou, and G. Ellinas are with the Department of Electrical and Computer Engineering and the KIOS Research and Innovation Center of Excellence (KIOS CoE), University of Cyprus, Nicosia, Cyprus. P. Kolios is with the Department of Computer Science and the KIOS CoE, University of Cyprus. Emails: {grigoriou.yiannis, nsouli02, chrysanthou.a.panagiotis, pkolios, gellinas}@ucy.ac.cy This work was supported by the Border Management and Visa Policy Instrument (BMVI), co-financed by the European Union and the Republic of Cyprus (BMVI/2021-2022/SA/1.2.1/015) (project REACTION). It was also partially supported by the European Union’s Horizon Europe research and innovation program under grant agreement No 101187121 (EUSOME) and by the Republic of Cyprus through the Deputy Ministry of Research, Innovation and Digital Policy. methods have been utilized in UAV applications for motion estimation and stabilization. By analyzing the motion of objects between consecutive video frames, optical flow techniques provide critical information on relative displacement [10], [11], delivering valuable information in environments where global navigation satellite systems (GNSSs) are unavailable or unreliable. In this work, an integrated system is implemented to achieve the two-fold objective of state estimation and trajectory prediction by combining information obtained from the DJI Matrice 300 RTK (equipped with advanced sensors), the DJI Zenmuse H20T, and the TriSonica Mini Wind and Weather Sensor. The Jetson Xavier NX embedded device (main processing unit) is used to collect the various sensor modalities using the DJI-OSDK. This configuration not only enables the collection of comprehensive flight data via the drone’s IMU and GPS but also incorporates a custom visual odometry algorithm that performs displacement estimation based on the geographical coordinates of the UAV agent’s home point. This approach enables the location estimation and state identification of the DJI Matrice 300 RTK in GNSSdenied environments. This research attempt extends our previous trajectory prediction and state estimation system (MELF-ST) as detailed in [12] as well as the multimodal sensor dataset described in [13]. The proposed research approach named MultiTask Learning Framework Feature Based Visual Odometry (MTLF-FVO) fuses numerous sensor data (e.g., IMU, GPS, weather data, and feature-based visual odometry outputs) along with a Transformer-based neural network architecture for trajectory prediction and state identification in GNSSdenied environments. The proposed MTLF-FVO system can provide accurate and robust position estimates in GNSSdenied scenarios, thus significantly enhancing UAV navigation reliability and operational safety. Specifically, this work’s contributions are: •The implementation of an innovative visual odometry algorithm for accurate and reliable displacement estimation in GNSS-denied environments. •The development of a transformer-based model for realtime trajectory prediction and state identification. •A real-world dataset that includes 3D complex flight paths and data from various sensors such as the drone’s IMU, weather data, and displacement estimation using feature-based visual odometry is designed to evaluate the proposed MTLF-FVO system. •The implementation of a system prototype (hardware and software) that is thoroughly examined in numerous outdoor experiments to evaluate its performance in trajectory prediction and state estimation. The remainder of the paper is organized as follows: Section II reviews related work, Section III provides an overview of the proposed multi-task learning framework, while Section IV details the visual odometry system. Section V describes the hardware setup and the process of dataset creation. Section VI presents the experimental evaluation and performance analysis of the system. Finally, Section VII concludes the work and outlines directions for future work. II. RELATED WORK Previous research efforts on UAV trajectory prediction using optical flow and visual odometry attempted to fuse optical flow and IMU data for UAV motion and velocity estimation using a combination of the inter-frame difference algorithm and Pyr-LK in a GNSS-denied environment [14]. In a similar vein, a solution of an algorithm for velocity estimation using Shi-Tomashi and Lukas Kanade tracker was implemented in [15]. Recent advances in visual odometry have significantly improved UAV trajectory estimation, particularly in challenging environments [16], showcasing a feature-based simultaneous localization and mapping (SLAM) system that estimates UAV trajectory in real time using ORB-SLAM. In addition, the work in [17] demonstrates a combination of deep feature extraction and visual odometry that further enhances the performance in environments with limited texture and poor light conditions. Furthermore, in [18], optical flow (OF) has been employed as a GPS-free technique for UAV navigation. This study presents the utilization of Lucas Kanade’s optical flow in conjunction with a Kalman filter to estimate velocity and altitude. Also, in [19] an autonomous quadrotor that harnesses a suite of sensors for navigation in GPS-denied environments is presented, illustrating the ability to use optical flow measurements to achieve 2D positioning, while infrared, inertial, and pressure sensors are incorporated to accurately determine the 3D position in real-time. In addition, the work in [20] investigates a multi-sliding window classification adaptive unscented Kalman filter (MWCAUKF) that leverages a timestamp-based data sorting algorithm to enhance positioning precision and stability. The proposed method operates in three phases, at first the fusion of multisensor data based on timestamps, then the estimation of measurement noise using sliding windows, and finally the application of data classification to mitigate filter instability caused by time-varying noise. Moreover, a previous research effort on trajectory prediction using neural networks (NNs), as presented in [7], illustrates an LSTM model that forecasts the UAV’s future location based on historical data (using a specified time horizon), presenting an improved accuracy in short-term trajectory prediction compared to conventional methods. Also, the authors in [21] examines the employment of a cylindrical antenna array to determine the UAV’s elevation and positional angles combined with an RNN model that predicts direction of arrival (DOA)-related angle data and the UAV’s location. Further, focusing on UAV state estimation using neural networks, the work in [22] introduces a deep learning framework that is used to enhance the estimation and accuracy of the UAV’s attitude and heading state. The main focus of this work was to estimate the UAV’s actual state by processing raw sensor inputs such as IMU sensor measurements. Equally important is the creation of a dataset that can capture the full range of environmental conditions, sensor variations, and operational scenarios necessary for training robust models. In particular, in machine learning applications it is crucial to develop a dataset capable of conveying all the parameters influencing the design of a reliable and robust NN architecture (such as environmental factors, sensor divergences, and operational contexts). For example, EuRoC MAV Dataset [23] is a widely used dataset created for evaluating UAV vision and state estimation algorithm, providing 11 benchmark flight sequences collected from a micro-drone in indoor rooms and an industrial hall, with accurate motion-capture ground truth. The sequences range from low-velocity flight paths with good lighting to fast agile maneuvers flight paths with poor illumination, enabling the development of SLAM, visual odometry, and sensor fusion methods. In another effort, a dataset focusing in high-speed drone flight paths with aggressive accelerations and rotations for trajectory or state estimation [24] was developed to provide stereo RGB videos, an event camera stream, IMU data, and precise ground-truth poses at 200 Hz. The proposed work (MTLF-FVO) complements and further extends the aforementioned research efforts by initally developing a UAV multi-modal sensor dataset (e.g., IMU, velocities, weather data, visual odometry measurements, etc.) [25] and subsequently examining the employment of a Transformer-based NN architecture (i.e., without requiring information from the GPS) to obtain real-time trajectory prediction and state estimation. Compared to the existing literature, the novelty of the proposed approach lies in the real-time UAV multi-task learning setting (location and state estimation extraction) in GNSS-denied environments, using sensor fusion methodology established on an onboard Transformer-based model. III. SYSTEM OVERVIEW A. System Architecture and Data Processing Pipeline The proposed system processes drone telemetry data in real-time using a modular architecture built around the Robot Operating System (ROS). As shown in Fig. 1, telemetry data is first captured from the drone through the onboard software development kit (OSDK). This data is then published to a dedicated ROS topic which streams real-time telemetry such as position, velocity, and orientation. Fig. 1: System overview. The data from this ROS topic is simultaneously stored in a database for future use and labeling. This labeling procedure (Alg. 1) categorizes each timestamp into modes such as “IDLE-HOVER”, “ASCEND”, “DESCEND”, “TURN”, or “HMSL” (horizontal movement straight line), depending on thresholds ∆z,∆ψ,∆happlied to altitude, yaw rate, and horizontal displacement. Algorithm 1 Flight Mode Labeling Input: Raw data entries 1: procedure LABEL FLIGHT MODES 2: Initialize mode =“IDLE-HOVER” 3: for each timestep in FlightData do 4: ∆z=zt−zt−1 5: ∆ψ=ψt−ψt−1 6: ∆h=p(xt−xt−1)2+ (yt−yt−1)2 7: if ∆z > ascend threshold and |∆ψ|<yaw threshold then 8: mode =“ASCEND” 9: else if |∆ψ| ≥ yaw threshold then 10: mode =“TURN” 11: else if ∆h≥horizontal threshold then 12: mode =“HMSL” 13: else if ∆z < −descend threshold then 14: mode =“DESCEND” 15: else if ∆z≈0and ∆h≈0and |∆ψ| ≈ 0then 16: mode =“IDLE-HOVER” 17: end if 18: end for 19: end procedure Output: Labeled data Subsequent to the labeling procedure, the structured time series sequence is then segmented using a sliding window approach to prepare input-output data for forecasting. These overlapping sequences capture the temporal dependencies required by the NN. The model is a Transformer-based architecture trained to jointly predict future trajectory points and classify flight modes (state estimation of the UAV agent). Afterwards, the multi-dimensional sequences are reshaped and normalized using the min–max scaling method and are then reverted to their original shape. The processed data are then utilized as inputs into a pre-trained open neural network exchange (ONNX) model, which outputs the multi-task objective (trajectory predictions and flight mode classifications). B. Transformer-based NN Model The proposed MTLF-FVO system incorporates a Transformer-based model that is designed to achieve the two-fold objective of trajectory prediction and state classification over a defined forecast horizon by employing a multi-task learning (MTL) framework. Specifically, a series of Transformer encoder blocks that combine layer normalization, multi-head self-attention, and feed-forward networks are employed to capture temporal dependencies in the input data. The Transformer-based network is comprised of two branches, one for forecasting continuous 3D trajectory outputs and one for binary classification. Both branches are enhanced with a positional encoding layer to preserve the input’s sequence order. By applying TimeDistributed dense layers and slicing the outputs to focus on the forecast horizon, the model can effectively compute future information related to trajectory prediction and state estimation. Also, the input data is structured as a window of 90 time steps, allowing the Transformer-based NN to consider 90 sequential observations to comprehend the UAV’s underlying dynamics. By employing the input data sequence (input data of 90 time steps), the model can achieve predictions over 30 time steps into the future (corresponding to a 3-second prediction). This design enables the network to leverage historical data effectively and compute both the continuous trajectory predictions and state classifications for a precise future interval, ensuring that short-term temporal patterns are accurately captured and projected. IV. FEATURE-BASED VISUAL ODOMETRY A. Feature-based Motion Estimation for UAV Navigation This section details the vision-based motion estimation framework developed for UAV navigation, leveraging sparse feature matching to compute relative displacements from consecutive video frames and thereby providing a realtime estimate of the UAV’s trajectory. The proposed visionbased framework utilizes efficient keypoint extraction and matching to achieve robust visual odometry on resourceconstrained embedded platforms. The feature-based visual odometry algorithm is embedded within the ROS framework and it operates at a pre-defined frequency of 10 Hz to ensure real-time performance. It should be noted that the sparse feature detection approach, based on feature detectors and binary descriptors, is utilized in order to minimize the computational overhead. However, this method introduces the accumulated error (sensor drift) trade-off over extended flights that may degrade positional accuracy. B. Camera Calibration and Intrinsic Parameter Computation Accurate motion estimation is primarily based on the camera calibration phase that computes the intrinsic parameters with the employment of the pinhole camera model [26], [27]. Given the horizontal and vertical fields of view (FOV), denoted by θhand θv, respectively, and the image dimensions (image width Wand image height H), the effective focal length (in xand ydimensions) in pixels is calculated as: fx=W 2 tan θh 2, fy=H 2 tan θv 2.(1) The principal point is assumed to be at the center of the image, leading to the intrinsic camera matrix K: K=  fx0W 2 0fyH 2 0 0 1  .(2) Therefore, the employment of the intrinsic camera matrix parameters enables the accurate transformation of imagespace measurements into real-world distances [26], [27]. C. Image Processing In this work, the UAV’s camera captures high-definition images at 1920×1080 resolution, which are then downscaled to 1280 ×720 to balance image quality with the onboard processing efficiency. Although the initial frame format is rectangular, the images are subsequently cropped into a circular frame as depicted in Fig. 2. A circular frame is utilized for the visual odometry calculation as it offers several advantages over rectangular or polygonal frames. For example, the use of a circular frame minimizes edge distortions by excluding the highly deformed peripheral regions that are often present in the corners of rectangular or polygonal images. Moreover, by focusing on a central circular region, the computational load is reduced without degrading data quality (rectangular and polygonal frames require additional processing steps, such as masking or filtering, to achieve similar results). (a) Rectangular Frame (b) Circular Frame Fig. 2: Feature extraction and matching. D. Sparse Feature Extraction and Matching Following a grayscale conversion, a vision detector is employed to extract feature keypoints from each incoming video frame. Afterwards, binary descriptors are computed from the frame keypoints to enable rapid matching using the Hamming distance method [28]. Consecutive frames are then compared by matching the computed descriptors from the current frame with those from the previous frame. A brute-force matching strategy is then employed. However, to counteract false positive correspondences, a robust estimation technique based on Random Sample Consensus (RANSAC) is used to estimate a homography matrix [27]. The homography matrix encapsulates both rotational and translational transformations between frames and is decomposed (with respect to the intrinsic camera matrix K) to isolate the translational displacement. E. Pixel-to-World Coordinate Transformation The average pixel displacement (∆x, ∆y)is determined with the employment of matched keypoints, and these values are then converted to real-world coordinates. With alt denoted as the UAV’s altitude, the FOV in physical units is computed as: FOVx= 2 ·alt ·tan θh 2,FOVy= 2 ·alt ·tan θv 2.(3) Also, the meters-per-pixel factors in horizontal (mx) and vertical directions (my) are determined as: mx=FOVx W, my=FOVy H.(4) Therefore, the translation from pixel to meter units ∆xreal,∆yreal is calculated as: ∆xreal = ∆x·mx,∆yreal = ∆y·my.(5) The pixel-to-meter transformation is critical for converting image-derived measurements into real-world navigation data. F. Compensation for Drone Orientation The computed displacement vector (in meters) must be aligned with the global reference frame. To achieve this, a rotation is applied to compensate for the heading angle θof the UAV. The transformation utilizes a standard twodimensional rotation matrix as described in Eq. 6 [27] to calculate ∆x′,∆y′: ∆x′ ∆y′=cos θ−sin θ sin θcos θ∆xreal ∆yreal.(6) This rotation ensures that the final displacement vector accurately represents the movement of the UAV with respect to a fixed geographic coordinate system, allowing for precise trajectory estimation. Fig. 3: MTLF-FVO hardware configuration: (a) DJI M300, (b) NVIDIA Jetson Xavier NX, (c) E-Port module. V. IMPLEMENTATION A. Hardware Configuration The DJI Matrice 300 RTK [29] alongside the DJI Zenmuse H20T [30], Nvidia Jetson Xavier NX [31], and LI-COR TriSonica Mini [32] are integrated to create the main platform for testing and data collection (Fig. 3). The main processing unit is the Jetson Xavier NX, due to its small size and ability to execute NN models onboard the UAV agent. Data collection is achieved using the E-Port Development Kit that enables data transfer from the UAV’s flight controller (with the employment of the OSDK) to the embedded processing unit. Also, the ROS is combined with the OSDK, creating a customized OSDK-ROS framework used to retrieve sensor data by utilizing designated ROS topics. B. Dataset Creation In the proposed ROS implementation, the DJI OSDK-ROS package is employed to subscribe to the drone’s telemetry topics and capture sensor data in real time. The implementation involves subscribing to standard telemetry messages as well as custom topics that are created to collect additional sensor information (incorporating external sensors such as the weather station and camera). Additionally, a featurebased visual odometry dataset is published via a dedicated topic, which includes visual odometry measurements and cumulative displacements that are then converted to GPS coordinates (latitude, longitude). The final topic includes a vast range of sensor features, as shown in Table III, that are then stored in a custom-designed dataset. In particular, the proposed dataset is generated from data collected during 18 different UAV flights over the course of 5 days, with the corresponding flight paths illustrated in Fig. 4. Also, Table I presents the weather conditions recorded on the data acquisition dates, providing a comprehensive overview of the environmental factors that may influence the collected measurements. TABLE I: Weather data Date Temperature (°C) Wind Speed (km/h) Humidity (%) 26/02/2025 13-14 6-11 48-51 28/02/2025 17-18 7-11 39-45 04/03/2025 17-19 30-35 40-45 13/03/2025 25-28 9-20 23-32 14/03/2025 27-29 7-15 20-26 The dataset is acquired using Real Time Kinematic (RTK) positioning which, as compared to the standard GPS, it improves accuracy by using carrier phase measurements and real-time corrections by employing a dedicated base station. This results in centimeter-level accuracy, offering a significant improvement in the precision of positioning data for the DJI Matrice 300 as shown in Table II. Therefore, the GPS+RTK measurements are used as the ground truth (GT) in this work to compare the proposed system’s accuracy in terms of localization. TABLE II: Positioning accuracy of the DJI Matrice 300 RTK under different modes Positioning Mode Horizontal Accuracy Vertical Accuracy GNSS (Non-RTK) ±1.5 m ±0.5 m RTK Fixed Solution 1 cm + 1 ppm 1.5 cm + 1 ppm C. Sensor Fusion Under the conditions of low vertical velocity, the proposed system leverages a sensor fusion method to combine velocity-based and feature-based visual odometry displacement estimates. The use of weighted factors for each source mitigates sensor-specific errors and achieves a more robust (a) Circular mission. (b) Triangular mission. (c) Linear mission. (d) Rectangle mission. (e) Multi-dimensional mission. Fig. 4: Data acquisition missions. cumulative displacement measurement. Therefore, the resulting fused displacement is then converted into geographic coordinates. This sensor fusion process provides a robust displacement estimate that enables accurate tracking of the UAV’s movement, even in scenarios where GPS signals are unreliable or unavailable. VI. PERFORMANCE EVALUATION A. Synthetic Dataset To generate a synthetic version of the dataset, controlled noise to the original data is applied. The process is performed on a per-flight basis to ensure that all data samples corresponding to a given flight are transformed consistently. The proposed approach involves the following steps: 1) Additive Noise on the Geographic Coordinates: For the xgeographic coordinates, the transformation is defined as x′=x+uwhere u∼ U(−5×10−5,5×10−5). The same offset uis applied to all data samples within a given flight group, preserving the relative spatial structure. 2) Multiplicative Gaussian Noise for the Remaining Dataset Features: For the ycoordinates, the transformation is defined as y′=y×(1 + ϵ)where ϵ∼ N (0,0.05). This scaling simulates realistic measurement variations. B. Experimental Results The MTLF-FVO model’s performance is evaluated in both trajectory prediction and classification accuracy. Two scenarios are considered as detailed in Table IV. The Transformer- TABLE III: Multi-modal data features employed to create the proposed dataset Features Measurement Unit Weather Data Wind speed Meters per second (m/s) Wind angle Angle of airflow (deg) Power Consumption Battery Voltage Voltage (V) Battery Current Amperes (A) GPS Data Latitude Decimal degrees (deg) Longitude Decimal degrees (deg) Altitude Meters (m) Orientation X, Y, Z, W Quaternions Velocity X, Y, Z Meters per second (m/s) Angular Velocity X, Y, Z Radians per second (rad/s) Linear Acceleration X, Y, Z Meters per second2(m/s2) Other Data Payload Kilograms (Kg) Attitude Yaw, Pitch, Roll, Heading Degrees Electronic Speed Controller Speed Round per minute Voltage Voltage (V) Feature-based Visual Odometry Latitude Decimal degrees (deg) Longitude Decimal degrees (deg) TABLE IV: Inference input types for MTLF-FVO model Label Description Fusion Inference using only sensor fusion (velocity + visual odometry, without RTK) NoLoc Inference without RTK or sensor fusion inputs and LSTM-based models are trained with a batch size value of 32,2transformer blocks, a feed-forward dimension of 1024,4attention heads, and a head size value of 64. Additionally, the dataset is partitioned into 70% for training, 15% for validation, and 15% for testing purposes. Initially, Table V presents the average Euclidean error for trajectory prediction based on the testing dataset and using a window size (WS) of 90 and a forecast horizon of 3seconds. The Fusion approach significantly outperforms the NoLoc model, showcasing the value of incorporating sensor fusion over relying only on telemetry data in GNSS-denied environments. These results demonstrate that generating latitude and longitude estimates through sensor fusion, leveraging both drone telemetry and camera data, substantially enhances predictive accuracy. TABLE V: Average Euclidean distance error for trajectory prediction using WS=90 and Forecast Horizon=3s All testing flights Prediction time (s) Euclidean error (m) MTLF-FVO Fusion Euclidean error (m) MTLF-FVO NoLoc t+1 4.10 11.50 t+2 4.09 11.50 t+3 4.20 11.35 Figure 5 illustrates the training and validation loss curves for the proposed Fusion configuration, showing a decreasing trend in both training and validation over a number of epochs that indicates the MTLF-FVO model is effectively learning in each scenario. Fig. 5: Training and validation loss of MTLF-FVO using sensor fusion inputs. Table VI reports the average Euclidean distance for trajectory prediction on the testing dataset in [25], evaluated using now a window size of 90 and a forecast horizon of 6seconds. The results showcase that the proposed MTLFFVO Fusion model outperforms all other models across all 6seconds. Even though LSTM RTK uses real-time kinetics and GPS information, it is still outperformed by the proposed MTLF-FVO Fusion which does not use RTK positioning. Furthermore, both NoLoc models result in higher errors compared to the RTK-enabled and fusion models. These results demonstrate that generating latitude and longitude estimates through sensor fusion, leveraging both drone telemetry and camera data, substantially enhances predictive accuracy. TABLE VI: Average Euclidean distance error for trajectory prediction using WS=90 and Forecast Horizon=6s All testing flights Prediction time (s) Euclidean error (m) MTLFFVO Fusion Euclidean error (m) MTLFFVO NoLoc Euclidean error (m) LSTM RTK Euclidean error (m) LSTM NoLoc t+1 4.10 11.50 7.62 17.05 t+2 4.09 11.50 7.55 15.86 t+3 4.20 11.35 7.51 15.46 t+4 4.92 11.83 7.51 15.42 t+5 5.37 12.02 7.73 15.63 t+6 5.72 12.25 9.13 16.83 Further, MTLF-FVO Fusion has been shown to produce less error and a significantly tighter error distribution compared to MTLF-FVO NoLoc, with the majority of errors concentrated below 5meters (see Fig. 6a). This observation is further reinforced by Fig. 6b, which presents the cumulative distribution function (CDF) of Euclidean errors. According to the CDF, 95% of the errors for MTLF-FVO Fusion for all time steps fall below approximately 5meters, whereas for MTLF-FVO NoLoc, the 95% lies around 11.5meters for all time steps, clearly demonstrating the superior accuracy and reliability of the Fusion approach. Although both models at t+ 1 achieve better overall performance, the differences between the three time steps remain relatively small. Additionally, Fig. 7 shows that MTLF-FVO Fusion achieves substantially lower mean absolute errors compared to MTLF-FVO NoLoc across all axes, with reductions from 4.02m to 1.59m in longitude, from 3.64m to 1.38m in latitude and from 1.35m to 0.55m in altitude. (a) Error distribution over time. (b) Average cumulative distribution function error at t+ 1,t+ 2 and t+ 3. Fig. 6: Comparison of Euclidean error performance for MTLF-FVO Fusion and MTLF-FVO NoLoc. Fig. 7: Per-axis position error (compared to RTK). In terms of state estimation, Table VII demonstrates that the proposed models achieve similar performance in terms of class-wise precision, recall, and F1-scores across most of the UAV classification states. It is shown that GPS and sensor fusion data are not necessary for effective state estimation. However, the UAV’s telemetry can provide highly accurate results, as demonstrated by the strong weighted average performance. For a comparison across various positioning algorithms, Table VIII compares their mean Euclidean error for a 3meter trajectory prediction. Among all models not based on GNSS, MTLF-FVO Fusion achieves the lowest error of (4.13 m) , outperforming the MTLF-FVO NoLoc (11.45 m) and the LSTM NoLoc (14.28 m). Even when compared to the LSTM RTK variant (8.31 m), which uses RTK, MTLFFVO Fusion performs significantly better despite relying only on onboard sensor fusion and vision. Notably, MTLF-FVO Fusion performs better compared to the LSTM (4.50 m) and BESTCC-CNN-LSTM (4.43 m) models showcased in [33]. It should be noted that both the MTLF-FVO and LSTM models are trained on the dataset introduced in [25]. TABLE VII: Class-wise metrics (Macro and weighted average are calculated considering all states). Model State Precision Recall F1-Score MTLF-FVO Fusion IDLE-HOVER 0.77 0.50 0.61 ASCEND 0.94 0.85 0.89 TURN 0.84 0.70 0.76 HMSL 0.81 0.86 0.84 DESCEND 0.27 0.93 0.42 Macro-avg 0.73 0.77 0.70 Weighted-avg 0.79 0.76 0.76 MTLF-FVO NoLoc IDLE-HOVER 0.81 0.50 0.62 ASCEND 0.95 0.93 0.94 TURN 0.83 0.83 0.83 HMSL 0.81 0.86 0.83 DESCEND 0.28 0.92 0.42 Macro-avg 0.74 0.81 0.73 Weighted-avg 0.80 0.77 0.77 TABLE VIII: Comparison of mean Euclidean errors Model Mean Euclidean Error (m) 3 second Trajectory Prediction RTK 0.01 LSTM RTK 8.31 LSTM NoLoc 14.28 LSTM [33] 4.50 BESTCC-CNN-LSTM [33] 4.43 MTLF-FVO NoLoc 11.45 MTLF-FVO Fusion 4.13 Moreover, Figs. 8a and 8b present a linear and complex mission, respectively, to directly compare MTLF-FVO Fusion and MTLF-FVO NoLoc. Clearly, the model operating without any GPS or sensor fusion input (NoLoc) performs with decreased accuracy in both linear and complex missions. In contrast, the MTLF-FVO Fusion model that employs sensor fusion of data achieves results comparable with the GT values. Finally, Fig. 9 presents the CPU, GPU, RAM usage, and power consumption of the system, with 20 W being the maximum power capacity of the Jetson device. The proposed system demonstrates consistent RAM usage, remaining below 40%, and utilizes the GPU only during the NN model inference. Also, power consumption remains relatively low, averaging around 40% of the device’s capacity, highlighting the model’s efficiency. CPU usage reaches nearly 100%, primarily due to the computational demands of the featurebased visual odometry and the real-time labeling algorithm running onboard (the observed spikes in CPU usage correspond to periods when the labeling process is active). VII. CONCLUSIONS AND FUTURE WORK This work presents a real-time multi-task learning framework that leverages Transformer-based NNs for simultaneous UAV trajectory prediction and state identification. A key contribution of this work is the integration of a featurebased visual odometry algorithm, fused with other onboard sensor data, to enable robust GPS-free localization. This fusion strategy allows the UAV to navigate and predict its trajectory reliably in GPS-denied environments, thus enhancing operational safety. The experimental results obtained through extensive outdoor flight tests demonstrate that the proposed model outperforms previous approaches in terms of prediction accuracy. Additionally, a custom dataset was developed to support the training and evaluation of the proposed system under real-world scenarios. (a) Trajectory prediction - linear path. (b) Trajectory prediction - complex path. Fig. 8: Trajectory prediction of different paths using MTLFFVO vs. GT values. Future research avenues include alternative NN architectures, to assess their suitability for real-time deployment. Research efforts will also focus on expanding the dataset with additional diverse flight patterns and further optimizing the visual odometry algorithm to enhance localization accuracy and system robustness. REFERENCES [1] O. M. Bushnaq et al., “Unmanned aerial vehicles (UAVs) for disaster management,” in Nanotechnology-Based Smart Remote Sensing Networks for Disaster Prevention, 2022, pp. 159–188. [2] A. Anastasiou et al., “Swarm path planning for the deployment of drones in emergency response missions,” in Proc. ICUAS, 2020. [3] A. Savva et al., “ICARUS: Automatic autonomous power infrastructure inspection with UAVs,” in Proc. ICUAS, 2021, pp. 918–926. [4] S. Rezwan and W. Choi, “Artificial intelligence approaches for UAV navigation: Recent advances and future challenges,” IEEE Access, vol. 10, pp. 26 320–26 339, 2022. [5] N. Souli et al., “An autonomous counter-drone system with jamming and relative positioning capabilities,” in Proc. IEEE ICC, 2022. [6] J. Zhang et al., “UAV trajectory prediction based on flight state recognition,” IEEE Trans. Aerosp. Electron. Syst., vol. 60, no. 3, 2024. [7] P. Shu et al., “Trajectory prediction of UAV based on LSTM,” in Proc. ICBASE, 2021, pp. 448–451. [8] Y. Zhang et al., “Recurrent LSTM-based UAV trajectory prediction with ADS-B information,” in Proc. IEEE GLOBECOM, 2022. Fig. 9: Allocation of system resources for MLTF-FVO. [9] P. Shukla et al., “Trajectory-prediction techniques for unmanned aerial vehicles (UAVs): A comprehensive survey,” IEEE Commun. Surv. Tutor., pp. 1–1, 2024. [10] N. Souli et al., “Real-time relative positioning system implementation employing signals of opportunity, inertial, and optical flow modalities,” in Proc. ICUAS, 2021, pp. 229–236. [11] Y. Zhang et al., “The use of optical flow for UAV motion estimation in indoor environment,” in Proc. IEEE CGNCC, 2016, pp. 785–790. [12] N. Souli et al., “A multi-task transformer architecture for drone state identification and trajectory prediction,” in Proc. DCOSS-IoT, 2024. [13] Y. Grigoriou et al., “Drone onboard multi-modal sensor dataset for complex outdoor scenarios,” Sep. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.13682870 [14] Z. Hou et al., “Fusing optical flow and inertial data for UAV motion estimation in GPS-denied environment,” in Proc. CCC, 2019. [15] V. Grabe et al., “On-board velocity estimation and closed-loop control of a quadrotor UAV based on optical flow,” in Proc. ICRA, 2012. [16] R. Mur-Artal et al., “ORB-SLAM: A versatile and accurate monocular SLAM system,” IEEE Trans. Robot., vol. 31, no. 5, 2015. [17] L. Yu et al., “A robust learned feature-based visual odometry system for UAV pose estimation in challenging indoor environments,” IEEE Trans. Instrum. Meas., vol. 72, pp. 1–11, 2023. [18] A. Miller et al., “UAV landing based on the optical flow videonavigation,” Sensors, vol. 19, no. 6, 2019. [19] N. Gageik et al., “An autonomous UAV with an optical flow sensor for positioning and navigation,” Int. J. Adv. Robot. Syst., vol. 10, no. 10, p. 341, 2013. [20] J. Peng et al., “UAV positioning based on multi-sensor fusion,” IEEE Access, vol. 8, pp. 34 455–34 467, 2020. [21] K. Xiao et al., “Trajectory prediction of UAV in smart city using recurrent neural networks,” in Proc. IEEE ICC, 2019, pp. 1–6. [22] M. K. Al-Sharman et al., “Deep-learning-based neural network training for state estimation enhancement: Application to attitude estimation,” IEEE Trans. Instrum. Meas., vol. 69, no. 1, pp. 24–34, 2020. [23] M. Burri et al., “The EuRoC micro aerial vehicle datasets,” Int. J. Robot. Res., vol. 35, no. 10, pp. 1157–1163, 2016. [24] J. Delmerico et al., “Are we ready for autonomous drone racing? The UZH-FPV drone racing dataset,” in Proc. ICRA, 2019. [25] Y. Grigoriou, N. Souli, and P. Kolios, “Drone onboard multimodal sensor and feature-based visual odometry dataset for complex outdoor scenarios,” Mar. 2025. [Online]. Available: https: //doi.org/10.5281/zenodo.15089283 [26] D. A. Forsyth and J. Ponce, Computer Vision: A Modern Approach. Prentice Hall, 2012. [27] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge University Press, 2003. [28] R. Szeliski, Computer Vision: Algorithms and Applications. Springer, 2022. [29] DJI, “Matrice 300 RTK DJI Enterprise,” Available at https://enterprise. dji.com/matrice-300/specs. [30] DJI, “DJI Zenmuse H20T,” Available at https://enterprise.dji.com/ zenmuse-h20-series/specs. [31] NVIDIA, “Jetson Xavier NX,” Available at https://www.nvidia.com/ en-us/autonomous-machines/embedded-systems/jetson-xavier-nx/. [32] Trisonica, “Compact Weather Sensor LI-COR TriSonica Mini,” Available at https://www.geotechenv.com/pdf/field supplies/li core trisonica mini.pdf. [33] G. Zhong et al., “Short-term 4D trajectory prediction for UAV based on spatio-temporal trajectory clustering,” IEEE Access, vol. 10, pp. 93 362–93 380, 2022.