Offsite evaluation of localization systems: criteria, systems, and results from IPIN 2021 and 2022 competitions

92 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 Offsite Evaluation of Localization Systems: Criteria, Systems, and Results From IPIN 2021 and 2022 Competitions Francesco Potortì , Senior Member, IEEE, Antonino Crivello , Soyeon Lee , Blagovest Vladimirov , Sangjoon Park , Yushi Chen , Long Wang , Runze Chen , Graduate Student Member, IEEE, Fang Zhao , Member, IEEE, Yue Zhuge , Haiyong Luo , Member, IEEE, Antoni Perez-Navarro , Member, IEEE, Antonio R. Jiménez , Han Wang , Member, IEEE, Hengyi Liang , Member, IEEE, Cedric De Cock , David Plets , Member, IEEE, Yan Cui , Zhi Xiong , Xiaodong Li , Yiming Ding , Fernando Javier Álvarez Franco , Senior Member, IEEE, Fernando Jesús Aranda Polo , Student Member, IEEE, Felipe Parralejo Rodríguez , Student Member, IEEE, Adriano Moreira , Senior Member, IEEE, Cristiano Pendão , Ivo Silva , Miguel Ortiz , Ni Zhu , Member, IEEE, Ziyou Li , Valérie Renaudin , Member, IEEE, Dongyan Wei , Xinchun Ji, Wenchao Zhang , Yan Wang , Longyang Ding , Jian Kuang , Xiaobing Zhang , Zhi Dou , Chaoqun Yang , Sebastian Kram , Maximilian Stahlke , Christopher Mutschler , Sander Coene , Chenglong Li , Member, IEEE, Alexander Venus , Student Member, IEEE, Erik Leitinger , Member, IEEE, Stefan Tertinek , Klaus Witrisal , Member, IEEE, Yi Wang , Senior Member, IEEE, Shaobo Wang , Beihong Jin , Fusang Zhang , Chang Su , Graduate Student Member, IEEE, Zhi Wang , Siheng Li , Student Member, IEEE, Xiaodong Li , Shitao Li , Mengguan Pan , Member, IEEE, Wang Zheng , Kai Luo , Ziyao Ma , Yanbiao Gao , Jiaxing Chang , Hailong Ren , Wenfang Guo , and Joaquín Torres-Sospedra Abstract—Indoor positioning is a thriving research area, which is slowly gaining market momentum. Its applications are mostly customized, ad hoc installations; ubiquitous applications analogous to Global Navigation Satellite System for outdoors are not available because of the lack of generic platforms, widely accepted standards and interoperability protocols. In this context, the indoor positioning and indoor navigation (IPIN) competition is the only long-term, technically sound initiative to monitor the state of the art of real systems by measuring their performance in a realistic environment. Most competing systems are pedestrian-oriented and based on the use of smartphones, but several competing tracks were set up, enabling comparison of an array of technologies. The two IPIN competitions described here include only off-site tracks. In contrast with on-site tracks where competitors bring their systems on-site—which were impossible to organize during 2021 and 2022—in off-site tracks competitors download prerecorded data from multiple sensors and process them using the EvaalAPI, a real-time, web-based emulation interface. As usual with IPIN competitions, tracks were compliant with the EvAAL framework, ensuring consistency of the measurement procedure and reliability of results. The main contribution of this work is to show a compilation of possible indoor positioning scenarios and different indoor positioning solutions to the same problem. Index Terms—Channel impulse response (CIR) positioning, evaluation, fifth-generation (5G) positioning, foot-mounted inertial measurement unit (IMU), indoor navigation, indoor positioning, pedestrian navigation, smartphone-based positioning, vehicle positioning. NOMENCLATURE 5G Fifth-generation technology standard for broadband cellular networks. AoA Angle-of-arrival. AoD Angle-of-departure. Manuscript received 8 May 2023; revised 25 August 2023, 14 November 2023, and 13 January 2024; accepted 15 January 2024. Date of publication 18 January 2024; date of current version 20 March 2024. (Corresponding authors: Antonino Crivello; Joaquín Torres-Sospedra.) Please see the Acknowledgment section of this article for the author affiliations and financial support. Digital Object Identifier 10.1109/JISPIN.2024.3355840 API Application programming interface. BLE Bluetooth low energy. CI Channel information. CIR Channel impulse response. C-SLAM Channel SLAM. EKF Extended Kalman filter. EMI Error mitigation. ENU East, North, Up. ESKF Error-state Kalman fitler. FFT Fast Fourier transform. FP Fingerprinting. © 2024 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 93 GNSS Global Navigation Satellite System. GPS Global positioning system. HTTP Hypertext transfer protocol. IMU Inertial measurement unit. INS Inertial navigation system. IPIN Indoor positioning and indoor navigation. KF Kalman filter. LiDAR Light detection and ranging. LLA Latitude, longitude, altitude. LLOP Linear line of position. LOS Line-of-sight. MAE Mean absolute error. MCC Maximum correntropy criterion. ML Machine learning. MU Moving user. NHC Nonholonomic constraint. NIST National Institute of Standards and Technology. NLOS Non-line-of-sight. NTP Network time protocol. PCA Principal component analysis. PDF Probability density function. PDR Pedestrian dead reckoning. PERSY Pedestrian reference system. PF Particle filter. pRRU Pico remote radio unit. QQ Quantile–quantile. RF Radio frequency. RSRP Reference signal received power. RSS Received signal strength. RTOA Relative time of arrival. SNR Signal-to-noise ratio. SPA Sum-product algorithm. SRS Sounding reference signal. TAE Time alignment error. TOA Time of arrival. ToF Time-of-flight. TRP Transmission reception point. UE User equipment. ULISS Ubiquitous localization with inertial sensors and satellites. UL-TDOA Uplink time-difference-of-arrival. UWB Ultrawideband. VO Visual odometry. WHIPP Wica heuristic indoor propagation prediction. WKNN Weighted k-nearest neighbours. ZUPT Zero-velocity constraint. I. INTRODUCTION THE purpose of the IPIN competition is to create an environment where methods and algorithms for indoor localization can be tested in a controlled environment as realistic as possible [1]. The idea is that localization systems described on papers are next to impossible to compare in a significant way, for a variety of reasons. First of all, the fact that each research group almost always works and tests the system in its own laboratory or nearby facility. Second, systems are often complex and their description may omit some relevant parameters or implementation details, making them impossible to reproduce. Third, but not least important, the infrastructure required to support positioning may not be fully replicated in a different location. Benchmarking based on public competition is one way out of these problems: research groups are invited to showcase their system in a way that makes it possible to compare it with other systems on an even ground. This article introduces IPIN competition’s outcomes for the 2021 and 2022 editions. The main contributions of this work in relation to previous ones [2],[3],[4],[5] are as follows. 1) The introduction of the EvaalAPI, an open source API, that allows off-site tracks to simulate the stressing conditions of an on-site track. 2) New tracks added that introduce new challenging scenarios for indoor positioning, such as Track 8: fifthgeneration (5G) in open plan office. 3) New environments and challenges are introduced to already existing tracks. 4) Description of state-of-art solutions. II. EVAAL FRAMEWORK The EvAAL framework is a set of criteria for defining how an indoor positioning competition should be set up. It was defined in 2014 based on the original EvAAL competitions [6]. During a competition, a number of teams compete according to a set of rules, which define a track. A competition may include a number of tracks, each centred on different types of devices and each with its own rules. Tracks can be on-site, with teams gathering in a physical place to run their working systems, or off-site, with no physical gathering and no physical devices involvedfromtheparticipatingteam.Foreachtrack,competitors run one or more trials during which the performance of their systems is measured. In on-site tracks, an actor walks along a predefinedpathwhilecarryingorwearingthecompetingsystem, which continuously estimates and records its position along the path. Reference points are marked along the path, and position estimation errors are measured for each reference point. Competitors have the opportunity to survey the environment, running testing trials on their own, before running (usually) two scoring trials on which the competition score is computed and the final competition ranking is established. The same applies to off-site tracks, with the main difference being that the competitors do not collect any data, neither for surveying the environment nor for participating in the competition. Data collection tasks are delegated to track chairs. All competitors have the same surveyed data and participate in the competition with the same information. In short, the EvAAL framework considers the following four core criteria: 1) natural movement of an actor; 2) realistic environment; 3) realistic measurement resolution; 4) third quartile of point Euclidean error; and the following four extended criteria: 94 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 1) secret path; 2) independent actor; 3) independent logging system; 4) identical path and timing. The criteria are defined and discussed in [6] and analyzed comprehensively in [1]. A. Integrating Multiple Diverse Tracks An IPIN competition is in fact composed of several independent competitions, called tracks. Each track has its own rules and purpose. Competitors can participate in one or more tracks. Tracks adhere to the EvAAL framework, though to different degrees. Tracks can be either on-site or off-site as follows. 1) On-site tracks are run in real-time, with trials consisting of a real device collecting sensor data and carried by a real person (an actor) walking along a predefined path previously unknown to competitors. The device, which runs software written by competitors, continuously estimates and records the current position. Estimates are collected at the end of each trial and handed to track chairs,whothen comparethemtoa groundtruthunknown to competitors. In an on-site environment, competitors are free to explore the site themselves and make surveys to tune their systems and discover the specifics of the competition area. Usually, this happens the day before the competition proper. During competition proper, scoring is computed on the best of two trials done on the same path. 2) Off-site tracks are done on recorded data rather than on real-time collected data. Track chairs collect sensor data at a given location and then provide them to competitors. Competitors run their software on the sensor data, estimate positions, and then hand those estimates to track chairs to be compared with the ground truth. In an off-site competition, competitors are provided with training trial data and/ortesting trial datasetstocheckthattheirsystem indeed works and tune it as long as they need. Then, they are provided with (usually) two scoring trial datasets,on which the scoring is finally computed. As an example, the flagship track of the IPIN competition has been the on-site Smartphone track, track 1 in the years 2014– 2019.Therulesofthistrackhave specifiedthatcompetitorsmust implement their solution as an app running on a commercial offthe-shelf smartphone, without communicating with the outside world. Only sensors embedded into the smartphone have been allowed, and the use of external devices has been excluded. B. Similar Competitions As it often happens, many people have had the same idea around the same time. In 2011, the first EvAAL competition was held in Valencia (ES) as part of UniversAAL project (FP7ICT-2009-4), with two more editions in 2012 and 2013. Microsoft indoor localization competition was also born in 2011, in association with the International Conference on Information Processing in Sensor Networks [7],[8],[9].TheMicrosoft competition’s aim was to muster different teams around TABLE I BASIC STATISTICS FOR ALL PAST IPIN COMPETITIONS the world using as many different approaches as possible, with few constraints. Measurements were taken with the competing systems staying still at a number of reference points, and no attempt at realistic movement or environment was made. The competition was held yearly until 2017. The PerfLoc Prize Competition was run in 2018 by the NIST (U.S.), while the Positioning Algorithm Competition was run in 2019 by the IEEE Communication Theory Workshop. Both competitionswerecentredonRFsystems[5].Thefirstresponder smart tracking competition is a big and ambitious effort funded (again) by NIST with $8M, most of which will be awarded to competitors. It was launched in 2022 and is planned to be completed end of 2023. C. Previous IPIN Competitions IPIN competitions started in 2014 in Busan, with a single onsite track based on smartphones. The first off-site track was run in 2015. Years 2020–2022 have not seen on-site tracks because of travel restrictions. Table Isummarizes the location, number of tracks, and number of competitors participating in on-site and off-site tracks during the history of the IPIN competition. III. INNOVATION IN 2021 AND 2022 EDITIONS The lessons learned after organizing the off-site edition of the IPIN Competition 2020 [4] were considered in the 2021 and 2022 editions, which both brought significant innovation as follows: 1) the introduction of the EvaalAPI in off-site tracks; 2) considering new challenges in existing tracks; 3) the birth of several new tracks introducing new localization technologies; 4) novel solutions for the different tracks. A. EvaalAPI The EvaalAPI interface is used to run off-site tracks. It was introduced in 2021 as experimental and established in 2022 for downloading testing trials and providing position estimates. The purpose of the EvaalAPI interface (see Section IV)is to make the results of off-site tracks closer to those of on-site tracks by removing some distortions that became apparent in 2015 when the first off-site track was introduced. Distortions POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 95 include, for instance, fixing positions afterward and smoothing scoring trajectory with future information. B. New Challenges in Existing Tracks This is a short summary of new challenges introduced to existing tracks in 2021 and 2022. Each is described in deeper detail in the following sections. Track 3: Smartphone, introduced in 2015, exploits the sensors of a smartphone. In 2015, the competition was based on static Wi-Fi fingerprinting. In 2016, user’s motion and other sensors were introduced. In 2018, the first very large scenario was introduced, a shopping mall. In 2021, device diversity and user diversity were introduced to the track. Track 4: Foot-mounted inertial measurement unit (IMU), introducedin2018,exploitsdatagatheredbymultisensorequipmentmountedonthefoot.ThePERSY sensor used in 2018–2020 [3],[4],[5] was replaced by ULISS in 2021 [10], as the latter is able to deliver 3-D inertial and magnetic, pressure, and GNSS data. Track 7: CIR, introduced in 2020, where an actor moves around a warehouse-like environment wearing a tag that regularly transmits UWB signals and CIR readings are gathered by anchors positioned around the area. In 2021, a second scenario without training was provided, where clutter elements from the first scenario were moved within the environment, allowing the assessment of the adaptability to changes in the environment. In 2022, training and evaluation data were collected by different agents and the EvaalAPI was adopted. C. New Localization Technologies This is a short summary of new localization technologies introduced in 2021 and 2022. Each is described in deeper detail in the following sections. Camera: Introduced in 2022, this is the successor to the on-site camera Track 2 run in 2019, but with a different approach. Training data are a set of photographs taken in an apartment, together with the shooting position and orientation; scoring data are more photographs taken in the same environment by a moving actor. Vehicle: Smartphone on vehicle, reintroduced in 2022, exploits sensors of a smartphone attached to a car’s dashboard. Training data and scoring data come from a car driving along GNSS-impaired areas and in underground parking. 5G: Positioning based on 5G technology standard for broadband cellular networks, introduced in 2022, exploits reference signals sent by 5G smartphones to four base stations installed on the ceiling of an open office.The base stations send measurements to a location server which estimates the user equipment (UE) location, a method called 5G networkbased localization. This track is intended to encourage the development of innovative algorithms for 5G positioning. D. Novel Solutions for the Competition Tracks The indoor positioning community was challenged with several tracks in 2021 and 2022. A total of 26 (2021) and 29 (2022) teams submitted their proposal to participate in the competition, but only 13 (2021) and 26 (2022) submitted their outputs to participate in the competition (see Table I). The short description of the proposed indoor localization solutions is available on the Evaal website [11]. All teams from the 2021 and 2022 editions were invited to submit an extended detailed description to be reported in this manuscript. Sections V–Xinclude descriptions from those teams that accepted the invitation. IV. EVAALAPI From 2015 to 2020, sensor data recorded by track chairs were timestamped and stored into a file which was then sent to competitors. Competitors would then send back the results some days later, by a common deadline. In time, we observed thatcompetitorsweremoreandmoreoftentreatingthechallenge more like an optimization problem than the emulation of an on-site trial. Specifically, we observed three main ways where optimization differs from emulation of on-site behavior, and usually provides more accurate results. One difference is that on-site trials are causal, meaning that estimates provided by competitors are necessarily based only on past sensor readings. This can make a big difference in estimation accuracy, which is important because real localization systems are indeed causal. Another difference is that on-site trials are one-shot, meaning that if something goes wrong in a trial, you can not just retry it. A real localization system cannot ask the user to go back and try again if it detects inconsistencies in its estimation results. The last notable difference is that on-site trials are run in real-time, meaning that estimation is timestamped when it is provided, analogously to a real localization system, which uses the position information as soon as it gets it, in order to provide a smooth experience to users. The introduction of the EvaalAPI interface in 2021 forced off-site tracks to a behavior, which was causal, one-shot, and real-time behavior, making them more similar to on-site tracks. In the following, the detailed working of EvaalAPI is discussed and some comparisons are made about off-site tracks, which switched to EvaalAPI. A. EvaalAPI Concept To make off-site track behavior more similar to on-site tracks, the first step is to force causal behavior by forcing the competing system to provide position estimates as it reads data from multiple sensors. This is obtained by defining an API for providing sensor data to the competing system and getting position 96 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 Fig. 1. High-level view of the Nextdata loop. Fig. 2. EvaalAPI server data logic. estimates from it. The API is implemented as a web service. The competing system runs a loop where it repeatedly reads data from multiple sensors and provides a position estimate. The EvaalAPI server reads sensor data from a file, provided by track chairs, where each row is timestamped and contains data from one or more sensors. The server writes position estimates obtained by competitors to a file, which is subsequently used by track chairs to compute the score of the trial. 1) Forcing Causality—The Nextdata Loop: The EvaalAPI server waits for a Nextdata (Horizon, Position) command [an HTTP request] from the competing system acting as a client; it answers the request with timestamped sensor data, which it reads from the data file. Figs. 1and 2illustrate the server loop. In each iteration of the main loop, the client sends a Nextdata command. Each carries a position estimate, which the client has computed on the sensor data received from previous Nextdata commands, and a requested time horizon, indicating how much sensor data the client expects as an answer to the request. TABLE II BEST SCORES IN METres (FIRST AND SECOND PLACE)FOR ON-SITE AND OFF-SITE SMARTPHONE TRACK BEFORE AND AFTER EVAALAPI WAS INTRODUCED IN 2021 This interface forces causal behavior because the competing system can base its position estimates only on past (in virtual time) sensor data; it can exploit no forward knowledge. 2) Forcing One-Shot—Nonreloadable Trials: In order to force one-shot behavior, each scoring trial can be run only once, i.e., it is nonreloadable. Each Track provides a number of testing trials, i.e., reloadable ones, that can be used at will by competitors to tune their system. In addition, it provides a few (usually two) scoring trials, which can be run only once, on which the score is computed, and the best one used for ranking the competitors. 3) Forcing Real-Time—Managing Timeout: In order to force real-time behavior, the virtual time is linked to the wall time. Virtual time is relative to the time stamped on each line of the sensor data file and to the horizon used in each Nextdata (Horizon, Position) request. EvaalAPI forces real-time behavior by slowing down virtual time with respect to wall time by a slowdown factor V(V≥1), to account for network delays, transmission bottlenecks and server response time. In practice, EvaalAPI implements a leaky bucketwith aratedefined bytheslowdownfactorandathreshold useful for compensating occasional brief networking disruptions; a timeout occurs when the bucket empties. B. EvaalAPI Implementation Source code, including a demo program written in Python, is available and licensed under a GNU Affero General Public License [12], which allows anyone to use, modify and redistribute it freely. C. EvaalAPI Experience Results from the competition have shown that EvaalAPI has made a difference. This is most clear when looking at results from tracks 1 and 3, given in Table II, which summarizes the best scores. Tracks 1 and 3 are the oldest and more stable ones. They are based on the same technology and are run in similar environments, that is, using sensors from a smartphone in big office environments. Notably, in 2019 tracks 1 and 3 shared the same location and even some of the reference points. During the five on-site smartphone competitions, winning teams have always obtained scores in the range from approximately 4 to 9 m, which is the same that happened in the first POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 97 three years of the off-site competition. In 2018, scores from the off-site track started to diverge, becoming much better than those of the on-site one. This was even more clear in 2019, when the preparation phase—choosing the path and taking measurements—was done by the same people in the same area for both the off-site and the on-site tracks. In 2021, with EvaalAPI, the off-site track results were back again to realistic numbers. In 2022, results were bad, apparently because the competition was more difficult with respect to 2021. In 2023, tracks 1 and 3 shared the same environment and the same sensors: results, yet to be published, show again a realistic alignment between them. V. TRACK 2: CAMERA (2022) This section describes track 2, which was based on camera (computer vision) and took place only in 2022. A. Track Description The widespread availability and the combination of sensing, computation, and communication capabilities make smartphones an attractive platform for indoor localization. The preferred localization approaches are influenced by factors, such as infrastructure availability, size, and type of the target indoor site, people’s movement characteristics, desired frequency, latency, and accuracy of the localization result. Image-based localization does not require the presence of specific infrastructure, can handle relatively large sites, and can provide orientation along with position estimation. Although it is possible to obtain centimeter-level accuracy for room or apartment-sized sites, in practical applications, achieving and maintaining similar accuracy in large public areas remains challenging.Amongthe relevantissuescontributingtothischallenge are variability in visual appearance over time, irregular motion patterns and the presence of dynamic objects. The aim of the track 2 competition is to test image-based indoor localization for pedestrians. The target site was two floors of an office building with a test area of about 50 m ×50 m per floor. Using a smartphone, we collected image and sensor data but focused on using only image data for localization mainly due to the off-site setting of track 2. The training data limited site coverage to simulate requirements for simplified collection procedures. The scoring trials’ datafeaturedreducedframerateand largermotion variabilityincluding stopping, sitting/standing and meandering. The reduced frame rate was partially motivated by a general preference for solutions with lower power consumption and partially by practical time constraints for scoring trials in settings with limited internet connection speed. B. Environment and Measurement Setup Track 2 used three floors of an office building. Data from the third floor (site 1) were used for training and were provided to competitors in advance. Data from the other two floors (site 2) were collected along the trajectories shown in Fig. 3and divided into training (plotted in green) and testing (plotted in yellow and Fig. 3. Data collected at track 2 evaluation site 2—floor 1 (top) and floor 2 (bottom). From the collected testing data trajectories, two (red) were used for scoring trials and the rest (yellow) were kept in reserve. red) sets. The training data were provided to the competitors on the day before the scoring trials. Two trajectories from the testing data (red trajectories) were used for the scoring trials on the competition day. To obtain ground truth labels, a backpack setup with a LiDAR (HESAI Pandar-QT) was used to scan the site and build a localization map. Later, images (640 ×480, 30 fps) and sensor data were collected using a smartphone (SM-N986 N). Training data were collected sequentially by several subjects walking along the hallway in a closed loop, holding the smartphone in their right hand in front of the body, with the rear camera facing forward. The recorded pose (longitude, latitude, floor, orientation) was the pose of the subject and a reasonable effort was made to keep a steady offset of the smartphone relative to the body. For training data, 43524 images were recorded along four closed trajectories per floor, combining the inner and outer sides of the hallway with clockwise and counterclockwise directions. For testing data, 5244 images were recorded along seven closed trajectories, keeping each trajectory on a single floor. Unlike the training data, the testing data trajectories included larger variations in walking speed (with stopping and sitting) and direction. Two test data trajectories were selected for scoring trials on the competition day. The image frame rate was reduced 98 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 Fig. 4. Team CamLoc system architecture. to 3 fps and reference points were selected for error evaluation. Scoring trial 1 had a length of 170 m, with 735 images and 80 reference points. Scoring trial 2 had a length of 136 m, with 750 images and 64 reference points. C. Description of Competitors (Camera) 1) Team CamLoc: In this competition, the team realized a visual localizing system based on an image retrieval algorithm and VO. The system architecture can be seen in Fig. 4.The overall process of the whole system can be divided into the following parts. a) Step 1 Build image descriptor database: To localize the smartphone with an image retrieval algorithm, an image descriptor database needs to be built in advance. PatchNetVLAD [13] model was used to extract the descriptor vector for each image in the training dataset and save them as files for the following image retrieval step. b) Step 2 Image retrieval: During the test, the PatchNetVLAD model was used to extract the descriptor of the query image online. The similarity between the query image and images in the database will be calculated. The approach finds the most similar image in the database and uses its ground truth as the pose of the query image. In order to speed up the process of similarity calculation, keyframes from the image descriptor database for every 20 images are selected. Since image retrieval is a type of nonincremental localizing algorithm, it is used to predict the starting point and relocalize, which can reduce the cumulative error from the following VO step. c) Step 3 VO: It is time-consuming and less generalizable for each test image to retrieve a similar image in the database. So, a frame-by-frame monocular VO is implemented to locate the smartphone in a faster and more robust way, which enables the system to track the smartphone even in an unknown environment. For the sake of getting high-quality feature points and their matching relationship, SuperPoint [14] and SuperGlue [15] models were used in Team CamLoc’s system. To make the monocular VO work effectively, the team needs to align the pose and estimate the scale factor. The poses predicted by VO are in the camera coordinate system, so the pose between the VO coordinate system and the LLA coordinate system were transformed with the help of the ENU coordinate system. Since VO is a type of incremental localizing algorithm, the problem of cumulative error is inevitable. To solve the problem, the system relocalizes with image retrieval for every Nframes. In practice, Nis set to 30. As for the scale factor, the prediction and ground truth on the training dataset were aligned with the Umeyama algorithm to get the estimated scale parameter. VI. TRACK 3: SMARTPHONE (2021 AND 2022) This section describes track 3, which was based on the use of smartphones and took place in 2021 and 2022. A. Track Description The objective of track 3 is to evaluate the performance of different integrated navigation solutions based on regular smartphone sensor fusion (WiFi, Bluetooth, and inertial, among others) in an off-site context. As done in the 2016–2020 editions [2], [3],[4],[5],[16], a data collection strategy and evaluation procedure has been followed. All data for track 3 has been collected with the Android app “GetSensorData” [17],[18], which records and stores all data coming from sensors available in the smartphone into a single text file, i.e., into a logfile. As usual, the dataset is split into three independentsubsets,namely,training,validation,andevaluation using ML terminology. A novelty introduced in 2021 was the evaluation through the EvaalAPI, renaming those subsets into training trials, testing trials and scoring trials, respectively. The first set is for calibration purposes and covers most of the evaluation area, containing several simple short single-floor tracks with several key points at relevant positions including initial, final and turns in the tracks. In the training trials, the trajectory between two key points is almost straight. The second set is for validation. It contains useful data for competitors to evaluate their systems with long trials covering multiple floors and well-known locations for the key points. Generally, testing trials only include a few key points and the user’s movement is not restricted to straight lines between two consecutivekeypoints.Inaddition,newareasmightbeexplored. Testing trials allow the competitors to evaluate the accuracy of their solutions as many times asthey wish, getting an assessment of the level of maturity of their solution. The third and last set is devoted to evaluation purposes, allowing competitors to have an independent external evaluation without ground truth data, and contains three multifloor very POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 99 TABLE III SMARTPHONES USED IN TRACK 3 (2021 AND 2022) DETAILING THE COMMERCIAL NAME,THE MANUFACTURER CODE (IF ANY), THE ANDROID VERSION,THE SENSORS USED,AND THE EDITION WHEN THE SMARTPHONE WAS USED long tracks. In contrast to the systematic data collection done in the previous trials, the scoring trials include realistic movements (e.g.,simulatingauserthatwasmessagingortakingaphonecall) and stops. Only three unlabelled scoring trials were provided to competitors in each edition. The accuracy score of a scoring trial corresponds to the 75th percentile of the sample error in compliance with the Evaal framework. This error is the 2-D positioning error plus a penalty of 15 m times the absolute difference between the current and the estimated floors. The team’s score corresponds to the best (lowest) score among the three scoring trials. The main difference introduced in 2021 with respect to the previous editions was the incorporation of several devices and users instead of collecting all data with the same device, i.e., track 3 chairs challenged the competitors with device diversity. In 2021, training and validation data were collected with five smartphones, which are detailed in Table III. The table illustrates the range of sensors considered in the competition. For evaluation in 2021, two scoring trials collected with a Samsung S7 by two different users and a scoring trial collected with a Samsung A5 2017 were provided. This device diversity feature was kept in 2022 but with a different and larger subset of phones, as shown in Table III, only a few phones were used in both editions. For evaluation in 2022, three scoring trials were provided, collected with the smartphonesBQAquarisX5Plus,SamsungA315G,andSamsungA5 2017, respectively. Giventhe largeamountofdataand diversityofsmartphones,a reasonable sampling frequency was set in “GetSensorData” for all sensors to record at 100 Hz in both editions, 2021 and 2022. In addition to the logfiles, georeferenced floor plans are provided to competitors. Those floor plans may be useful at the sensor fusion level, allowing competitors to check whether the provided positions are coherent with the environment. The logfiles, supplementary materials and full technical descriptions are available in [19] and [20]. This package complements the ones from the previous editions [21],[22],[23], [24],[25]. B. Environment and Measurement Setup (2021) Forthe 2021edition,competitiondatacame from thefacilities of the University of Extremadura (Badajoz, ES). The collection lasted four days and was restricted to the external car park area, Fig. 5. Floor plan of track 3 (2021) environment and its auditorium (located on the bottom-right corner in floor 1). the ground floor and the basement. More than 30 BLE beacons were deployed in part of the environment to support indoor positioning, and their location was provided to competitors. The indoor area included an auditorium, which covers a large area with a soft floor transition as shown in Fig. 5. C. Environment and Measurement Setup (2022) In 2022, competition data came from the facilities of the University of Minho (Guimarães, PT). The collection lasted four days and was restricted to the School of Engineering, a threestorey building, and its surroundings. This time, no additional infrastructure was deployed to support indoor positioning. In addition, the scoring trials were collected one month after the training and testing trials. The indoor area is a three-storey variable-height building and it includes a large open patio as shown in Fig. 6. D. Description of Competitors (Smartphone) 1) Team Leviathan: The proposed system consists of the following four components. 1) PDR system based on step detection and stride length estimation. 2) ESKF incorporating IMU measurements, headings and PDR output. 3) Floor detection and initialization based on Wi-Fi fingerprint and barometer. 4) PF that utilizes the floor plan information. The flow chart of the proposed system is shown in Fig. 7. a) Pedestrian Dead Reckoning: The PDR algorithm estimates the pedestrian step count, stride length and heading. Thus, PDR exploits accelerometer, gyroscope, and magnetometer data. To remove high-frequency noise, a low-pass filter is 100 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 Fig. 6. Floor plan of track 3 (2022) environment and its patio. Fig. 7. Flow chart of the indoor localization solution proposed by team Leviathan. applied to the norm of the acceleration. Team Leviathan first employ FFT to convert acceleration data from the time domain into the frequencydomain to better represent the periodic component in the signal [26]. Then, detection is used to identify the steps. Unliketraditional stridelengthestimation,Leviathan’s approach is more adaptive in the sense that it is formulated as a function of the peak frequency [27]. The heading of the pedestrian is initialized and updated by using the Madgwick method, which combines magnetic field and angular velocity [28]. b) Error-State Kalman Filter: In the scenario of indoor localization, the types of sensors used may be different. An EKF is well-known to fuse different kinds of observations together. Compared with EKF, ESKF applies optimization based on the errorstate,whichisnumericallysmall.Thus,theestimationerror is smaller during the linearization process, leading to a more accurate result. In the prediction step, the gyroscope and acceleration are used to estimate the current pose. The magnetometer and the PDR result are used to correct the pose estimation. Specifically, the PDR module measures the displacement, and the magnetometer measures the current heading. The observation error is assumed to follow a Gaussian noise distribution to correct the pose estimation. c) Floor Detection and Initial Position and Pose Estimation: The Wi-Fi RSS fingerprint and the barometer are used for floor detection, initial position estimation, and position correction. RSS is susceptible to various environmental changes, e.g., concrete walls, moving humans, temperature and humidity [29]. Compared with PDR, Wi-Fi localization is less accurate. Thus, Wi-Fi RSS is mainly used for position correction and floor detection. A radio map is built from the training data, and the first few Wi-Fi detections are used to locate the initial pose. The variance of the barometer is used to detect the floor change by setting a threshold. The current pose is continuously matched with Wi-Fi location and floor information. Once a significant mismatch is detected, the system resets. Data from the accelerometer and the magnetometer are used to extend positions to pose estimations by adopting the Madgwick filter. d) Particle Filter: The PF fuses the trajectory estimated by the ESKF and compares it with the floor plan to regulate the distribution of particles. The floor plan images are first stored as an obstacle probability map. The floor estimation component first identifies the floor ID and an initial position guess. Each point xon a 2-D plane can be assigned with a Gaussian probability distribution L(x)= 1 σ√2πexp −minx∗∈Ωobs d(x,x∗) 2σ2(1) where Ωobs is the set of obstacles, x∗is the location of the nearest obstacles,andd(•)istheEuclideandistancebetween twopoints. During each update, a single particle infers its position and the associated probability of hitting an obstacle. In the resampling stage, particles in unreachable areas are removed. Moreover, to account for history information, a particle will be removed if its accumulated penalty within the time window exceeds a predefined threshold. New particles are generated following the distribution of valid particles. The final position is calculated as the weighted average of the particle positions. In case of system failure, i.e., when all particles hit obstacles or a wrong floor IDis reported by the floor detector, random particles will be generated on the specific floor until system convergence. 2) Team imec-WAVES 2021 and 2022: Team imecWAVES’ systemsforthe2021 and2022competitionsare similar and consist of six modules. Each module has the same functionality in both versions, but some modules are implemented differently. Fig. 8shows how these six modules and their components interact. The following paragraphs briefly describe each module. The description applies to both years unless specified otherwise. a) EvaalAPI interface: This interface starts the Evaal API trial and requests the next stream of smartphone data in blocks of 0.5 s. It parses and structures the received data and passes it to the PDR module, which employs a step detection algorithm (see Section VI-D2b). If the PDR module detects one or more steps, the interface waits for the path estimation algorithm to provide a new position. If no step is detected, the interface will take the previous position estimation. The position is sent back to the EvaalAPI server, and a new block of data is received. POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 107 IX. TRACK 7: CIR IN WAREHOUSE This section describes track 7, which was based on the use of CIR in warehouses and took place in 2021 and 2022. A. Track Description RF positioning in cluttered indoor environments is challenging. As signals travel through the environment along different pathsit is difficult to determine the correct ToF of the transmitted signals. Traditionally, fingerprinting-based solutions have been usedtoestimatearoughpositionfromnarrow-bandsignals,such as Wi-Fi or bluetooth. However, with modern UWB technology, signals can be transmitted at higher bandwidths, enabling a much higher spatial resolution from which complex propagation conditions can be extracted, such as absorption, reflection, diffraction, and scattering [43]. While UWB is progressively integrated, but not yet widely spread, into consumer devices, current progress in development and standardization makes it likely that it will be ubiquitous in the near future. This allows for low-cost ad hoc positioning. To leverage the benefits of the high spatial solution we can make use of the CI. For sufficiently high bandwidths the CI roughly corresponds to the complex-valued CIR. Many algorithms have been investigated that exploit the CIR to extract spatial information in order to enhance the positioning performance.Theyhavebeen usedforToF errormitigation[44],which uses the CIR to estimate an environment-specific ToF error, fingerprinting [45], which exploits the raw CIR as location-specific information, and C-SLAM [46], which exploits the multipath components included in the CIR. Therefore, besides ToF estimates, we provide the raw CIRs, which allows enhancement of localization accuracy. The challenge is divided into two parts. In the first part, the data that is used for training and testing originate from the same environment setup. In the second part, we made some changes to the environment setup (i.e., we moved mobile metallic objects) in order to consider the robustness of the algorithms to environmental changes. For the second scenario, we do not provide trainingdatabutonlytestdata.The trajectories we usefortesting in the second scenario stay within a similar area as the one used in the first scenario. In2022, we investigatedthegeneralizationtoa different agent forcollectingdata.Ina typicalindustrialapplicationsetting,data points are easily collected and labelled by automated guided vehicles, but the tracking targets can be other agents, such as persons. The different agents have various influences on the signal, due to the shadowing of, e.g., a person or reflections of a robot. Also, movement patterns and height of the radio unit are different, which might also influence the performance. The majority of the provided data for the validation and training are collected by a mobile robot, while the evaluation is based on the tracking of a worker in an industrial setting. In both years the ground truth of the transmitter positions is collected using a millimeter-accurate Qualisys motion tracking system. The data are collected and synchronized by an NTP server and preprocessed (corrupted data points are removed and RF and positioning reference data are synchronized). Fig. 20. Schematic environment setup, including exemplary object setups 1 and 2 (left-hand side) and a similar real-world environment (right-hand side). B. Environment and Measurement Setup (2021) The environment consists of an area of ≈300 m2, partially enclosed by reflecting walls (consisting of the walls of the measurement wall, including metal gates and artificially included reflector/absorber walls elements) and various metal objects that are typical of industrial indoor environments like, e.g., industrial vehicles or metal shelves. Fig. 20 (left-hand side) schematically sketches the environment, while the real-world environment is shown on the right-hand side. The receiving anchors are placed around the recording area at about 1.5 m height. The transmitter device is carried by a human/worker and regularly transmits UWB signals received by the anchors. The data are recorded using a platform based on the Decawave DW1000 UWB chip with a centre frequency of 4GHz and 499.2 MHz bandwidth. This challenge contains the following two scenarios. 1) For the first scenario a training dataset with ground truth positional information is provided; the models submitted by the competitors are evaluated on a test set (a few trajectories) that originates from the same measurement campaign, i.e., training and test datasets were recorded on the same environmental setup. Both training and test datasets contain complete trajectories while the trajectories of the test dataset are shorter. The testset does not contain ground truth position labels. 2) The second scenario presents a modification of the first scenario. In this setup, clutter elements within the environment (e.g., forklift, van, etc.) were moved, which led to a slightly different propagation scenario. The goal of this scenario is to test if the models submitted by the competitors overfit the previous environmental setup and fail to generalize well to changes to the environment. Therefore, the training dataset does not include any trajectory collected within this modified scenario, which is only considered in the trajectories within the testset. C. Environment and Measurement Setup (2022) The environment consists of a warehouse area of ≈1200 m2 partially enclosed by reflecting walls (consisting of the walls of the warehouse, including metal gates). The environment contains various metal objects (e.g., industrial vehicles or metal shelves). Fig. 21 shows a picture of a part of the warehouse. Receiving anchors are placed around the recording area at about 108 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 Fig. 21. Image of the environment. The mobile robot can be seen on the right. Fig. 22. Flowchart of the imec-WAVES localization approach. Each rectangular box represents a step in the process. Training the regression model for range error correction is done separately. 1.5 m height. The transmitter device is carried by the mobile agent/tracking target and regularly transmits UWB signals received by the anchors. In the data collection phase, it is attached to a mobile robot. In the evaluation phase, it is carried as a handheld by a human/worker. An exemplary and representative evaluation/experiment dataset for adjusting models was provided. D. Description of Competitors (CIR in Warehourse) 1) Team imec-WAVES (2021): imec-WAVES’ localization solution’s core is 1) distance estimation between each tag and anchor, 2) range correction through a regression model, and 3) a PF for localization. Fig. 22 shows the individual steps of the system. A custom ranging algorithm provides range estimates for each captured CIR. A first pass of a PF provides an initial approximation of the user’s location throughout time, from which their motion trajectory is calculated. Erroneous range estimates are identified by examining time-range plots for each tag–anchor pair. At this stage, a set of predictors serves as input for the regression model. Theregressionoutputsa scalardistancecorrectionthat isapplied to the original range estimate. In training, this reduced the MAE from 24 down to 3 cm (excluding uncorrectable estimates). A second PF utilizes the updated range estimates to obtain a better location estimate. Finally, a smoothing step in conjunction with position interpolation predicts the location at each requested timestamp. Fig. 23. (a) Normalized histogram fitting of the ToF estimation errors and (b) the QQ plots. a) Ranging algorithm: This solution calculates range or ToF using a threshold-based algorithm. The threshold is calculated from the noise, which is determined heuristically afterpartitioning the CIR into a noise region,a region-of-interest which contains the first path component, and a region with only multipath components and noise. b) Regression model: The correction model uses Gaussian process regression with a constant basis function and exponential kernel. Training is performed with fivefold cross-validation to reduce the risk of overfitting. Predictors are calculated from the following. 1) The original CIR measurement (anchor number, first path bin power, maximum bin power, and distance estimate). 2) The location estimate [AoD from the anchor and compatibility with distance estimate]. 3) The motion trajectory [velocity, direction, turning rate, and angle-of-arrival (AoA) on the tag]. c) Tracking: In this competition, tracking was leveraged to the PF algorithm, which has better accuracy than the Kalmanframework filters generally [47]. In this competition, only the 2-D coordinates PMU of the moving user (MU) are considered, which are updated via the constant velocity motion model, given as follows [48]: P(t+1) MU v(t+1) MU =I2Δt·I2 0I 2P(t) MU v(t) MU +Δt·12×1 12×1nv(3) where vMU denotes the 2-D velocity of the MU, Δtthe time difference between timestamps tand t+1. nvrepresents the Gaussian velocity errors. In total, 1000 particles were utilized to generate the proposed likelihood of the MU locations within the targeted area. The weights were updated via the PDF of the ToF estimation errors. However, in the case of NLOS propagation, the ToF estimates may have large offsets due to the wrong threshold judgment. To better quantify the ToF statistical error, its histogram was fitted on three widely-used distributions, namely, Gaussian, Laplace, and t location-scale distributions [49].Fig.23 shows the histogram fitting on these three distributions and their goodnessof-fit via QQ plots. Benefiting from handling with heavier tails, t location-scale distribution presents the most straight line in QQ plots, which illustrates that the ToF estimation errors best follow t location-scale distribution. POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 109 Fig. 24. Flowchart of the imec-WAVES localization approach. Each rectangular box represents a step in the process. Training the regression model for range error correction is done separately. The same implementation and parameters are used in both instances of the PF in the systems developed by team imecWAVES. 2) Team imec-WAVES (2022): In 2022, the localization solution’s core remains similar to the approach used by Team imec-WAVES during the IPIN 2021 competition. The main difference was that in 2021 the EvaalAPI interface was not used, sothesolutionpresentedin2022byimec-WAVESwasreal-time. It consists of the following. 1) Distance estimation between each tag and anchor. 2) Range correction through a regression model. 3) Two instances of a PF for localization. Fig. 24 shows the individual steps of the system. A custom ranging algorithm provides range estimates for each captured CIR. A first pass of a PF provides an initial approximation of the user’s location throughout time, from which their motion trajectory is calculated. A set of predictors serves as input for the regression model. The regression outputs a scalar distance correction that is applied to the original range estimate. A second, final PF utilizes the updated range estimates to obtain a better location estimate. The location estimate at each requested timestamp is calculated by using the prediction step of the final PF. a) Ranging algorithm: A threshold-based algorithm is used to calculate range or ToF. The threshold is calculated from the noise, which is determined heuristically after partitioning the CIR into a noise region, a region of interest which contains the first path component, and a region with only multipath components and noise. Algorithm parameters are obtained through the optimization of the training data. A fixed bias correction term is obtainedfrom the median ranging error of each anchor’s training data range estimates. The term is subtracted immediately after the ranging step. Negative range results are dropped. Previous range estimations are not used. b) Regression model: The correction model uses Gaussian process regression with a constant basis function and a Matern 5/2 kernel. Training is performed with fivefold crossvalidation to reduce overfitting. Predictors are calculated from the following. 1) The original CIR measurement (anchor number, first path bin power and maximum bin power). 2) The location estimate (AoD from the anchor, compatibility with distance estimate, Xand Y-coordinates). 3) The motion trajectory (velocity, direction, and AoA on the tag). Due to the real-time character of the data in 2022, trajectory estimation was much harder. In fact, predictors used in 2021, Fig. 25. QQ plot of the measurement likelihood functions that are used in the first and second PF. The first filter incorporates a function based on a Stable distribution fitted to unbiased ranging error. The second filter makes use of a t location-scale distribution fitted to the ranging error after it is corrected by the regression model. such as turning rate, proved too unreliable this time around. Furthermore, the regression model could only be trained with data that contained global timestamps, as these timestamps are needed to calculate the motion trajectory. This made the largest available set of training data not applicable for training this model. c) Tracking: PF algorithm was used to track the agent. Only the Xand Y-coordinates of the MU are considered. The particles are updated through a constant velocity motion model, using nonadditive process noise. The first PF contains 700 particles, with a measurement noise of 0.17 m and a process noise of 10 ms−2. The second PF contains 2000 particles, with a measurement noise of 0.02 m and a process noise of 15 ms−2. The measurement likelihood function to update the particles makes use of a PDF fitted to the range estimation errors from the training data. Fig. 25 shows the QQ plot of each distribution fit. For the first filter, the PDF is fitted using a stable distribution on the debiased estimates. In the second filter, a t locationscale distribution is fitted to the ranges after correction of the regression model. 3) Team SPSC: Team SPSC used a two-step method similar to the algorithm presented in [50],[51]. First, a snapshot-based parametric channel estimation and detection algorithm extracts delays and corresponding amplitudes of multipath signal componentsoutofthereceivedbasebandsignal.Second,asequential estimation algorithm estimates the state of the mobile agent by using the delays and amplitudes as measurements. More specifically,thesequentialestimationalgorithmjointlyperforms probabilistic data association and estimation of the mobile agent state together with all relevant model parameters, employing the SPA on a factor graph. It adapts in an online manner the timevarying component SNR as well as the detection probability of the LOS component. The concept of probabilistic data association, together with adaptation of the LOS detection probability, enables the algorithm to solve the nonlinear positioning problem and mitigate NLOS situations, while still offering an execution timeinthe magnitudeorderofmilliseconds.In thefollowing,the probabilistic system model of Team SPSC’s algorithm is briefly discussed. a) Channel Estimation and Detection Algorithm: Thechannelestimationanddetectionalgorithmpresentedin[51, Supplementary Material] was applied to the baseband signal vector at each time nand for each anchor jindependently. It provides a measurement vector z(j) ncontaining a number of 110 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 M(j) nmeasurements z(j) m,n =[ˆ d(j) n,m ˆu(j) n,m]T. Each z(j) m,n contains a distance measurement ˆ d(j) n,m, and a normalized amplitude measurement ˆu(j) n,m. b) SPA-based Sequential Estimation Algorithm: The components of the measurement vector z(j) nare subject to data association uncertainty, i.e., it is not known which measurement originates from the LOS, from multipath or from a false-alarm. Based on the concept of probabilistic data association, an association variable is defined as a(j) n=m∈{1...M(j) n},z(j) n,m is the LOS meas. in z(j) n 0,there is no LOS meas. in z(j) n. (4) It differentiates between the conditional likelihood functions for LOS and NLOS measurements, which, for the distance measurements ˆ d(j) n,m are given as a Gaussian PDF with mean value that is geometrically related to the agent position pnand a uniform PDF, respectively. The system jointly performs sequential estimation of amplitude states u(j) n, which are assumed to be independent per anchor. The corresponding conditional amplitude likelihood functions are given as Rician PDF and Rayleigh PDF for LOS or NLOS measurements, respectively. The LOS detectionprobability,whichoccursaspartofthedataassociationprior and represents the probability that there is a LOS component per time step and anchor, is modeled as the product pD(uj n)q(j) n, between the amplitude-related detection probability pD(uj n)and a prior LOS probability q(j) n. The latter is modeled discretely, as a first-order Markov process. The likelihood functions, together with the prior PDF of the data association variable, define the joint pseudolikelihood function ˜gz(z(j) n;pn,u (j) n, a(j) n,q(j) n). The agent state is described by the state vector xn= [pT nvT n]T, which is composed of the 2-D agent position pn and velocity vn. The agent motion, i.e., the state transition PDF Υ(xn|xn−1), is modeled by a linear, constant velocity, and stochastic acceleration model with standard deviation set to 1/3 of the mean step width of the mobile agent. The state transition PDF of the normalized amplitudes Φ(u(j) n|u(j) n−1)is modeled as a Gaussian distribution with standard deviation set to 5% of the last amplitude estimate. The elements of the first-order Markov transition matrix Ψ(q(j) n=ωi|q(j) n−1=ωk), as well as the initial distributions, i.e., f(x0)p(q(j) 0)f(u(j) 0),wereinitialized heuristically as described in [50]. By applying Bayes’ rule as well as some commonly used independence assumptions, the factorized joint posterior PDF is computed, which is visually represented by the factor graph shown in Fig. 26. The agent state is estimated as the minimum mean-squared error estimate given as ˆ xMMSE nxnf(xn|z)dxn.(5) In order to obtain (5) the marginal posterior PDF is calculated by performing message passing on the factor graph in Fig. 26 Fig. 26. Factor graph representing the factorization of the joint posterior PDF and the messages according to the SPA. See [50] for further details. Fig. 27. Measurement environments. utilizing the SPA rules. Since the integrals involved in the calculations of the messages cannot be obtained analytically, a sequential particle-based implementation is used. X. TRACK 8: 5G IN OPEN-PLAN OFFICE (2022) This section describes track 8, which was based on 5G in an open-plan office and took place in 2022. A. Track Description Track 8 was dedicated to 5G positioning based on UL-TDOA, which is widely adopted in 5G products. 2022 was the first time such technology was part of an IPIN competition. The Huawei 5G system is deployed in an indoor office in the Huawei–Chengdu building. The area is about 15 m×15 m with ceiling height of 3.2 m. There are working tables, chairs, and partition panels in the room with heights in the 0.5–1.5 range. Four pRRUs with known locations are mounted on the ceiling, see Fig. 27. The UE is a Huawei Mate 30 Pro terminal. The UE transmitsinburstsof80ms.ThepRRUsdetectsthereceivedSRS and calculates the positioning measurements, such as RTOA and RSRP. The UE was fixed onto a trolley with a constant height of 1.2 m, and the UE moves at a speed of 0.2–0.5 ms−1within the reachable area (highlighted in green colour). During the walking route, the UE signals to some TRPs might be blocked by tables, partition panels and shelves. The tables and panels are made of plywood (2–4 cm thick), and the shelves are made of sheet metal. Hence, there may exist a mixture of LOS, near LOS, POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 111 TABLE V COORDINATES OF THE PRRUS Fig. 28. Pipeline of team mobile’s solution. and NLOS channels. Strong multipath effects may also exist due to reflections from the environment, such as concrete walls, columns, and other metallic objects. Four sets of data are given, named Testing_A, Testing_B, Scoring_A, and Scoring_B. Each set contains 1000 measurements (≈85 s long) with 50 ground truth positions. Ground truth positions are only given for the testing sets. The RTOAs is measuredbyusingtheMUSICalgorithmwithaknownaccuracy of 1 ns tested in a LOS environment. There are existing timing errors among the receivers in TRPs, called TAE. The TAEs of the TRPs are unknown, but should be in the−100 to 10 ns range. ThecoordinatesofthepRRUs’slocationsaregiveninTableV. The ground truth coordinates of pRRUs3 are actually (12.48 and 9.75m),thatis,thexy coordinatesareexchangedwitheachother. This is to introduce a coordinate error, which might happen in practice. 1) Competition Area: For IPIN 2022, due to movement restrictions, the data set of track 8 was collected only from one indoor office instead of two independent indoor scenarios as planned. Fortunately, the measured office has diverse furniture and facilities, which enable diverse channels in different locations, such as strong LOS, near LOS, and NLOS. Four datasets are measured in the office, and their routines are different in time. Competitors were encouraged to develop self-localization for UL-TDOA including TAE estimation and pRRUs selection, possibly using artificial intelligence. B. Description of Competitors (5G in Open-Plan Office) 1) Team Mobile: The technical route used by mobile team is mainly based on an ML approach. Fig. 28 shows the pipeline of team mobile’s proposed solution. First, simulated data is generated based on data statistics for the purpose of data augmentation, the training data set is built and then CatBoost [52] is used to complete the end-to-end positioning task. Then the Fig. 29. (a) Relationship between the RSRP received by pRRU0 and the real distance. (b) Relationship between the difference of RSRP and the real distance difference of pRRU0 and pRRU1. position estimation is further corrected based on the room layout and spatio-temporal continuity of the trajectory. Finally, KF is used to smooth the trajectory and resample it in real-time. a) Data Analysis and Preparation: The data provided by Track 8 include the timestamps and eight features which are the RTOA and RSRP estimated by the four pRRUs. Note that there are timing errors among the receivers in pRRUs, called TAEs. The unknown TAEs greatly affects the accuracy of TOAs and the position estimation results, and it is also time-varying among different datasets. In contrast, RSRP is more stable because data in all data sets are collected in the same environment. In this case, the solution provided by Team Mobile identifies the RSRP as the key feature. To fully capture the RSRP feature, an ML-based approach is planned to be used in order to derive the user position. However, the labeled dataset provided by track 8 is too small. Thus, an interpolation method is applied to augment the labeled data. Then, the relationship between the RSRP received by each pRRU and the distance in the real data are quantified. Also, the relationship between the RSRP difference and the real distance difference of each pRRUs is evaluated. As shown in Fig. 29, both are negatively correlated. Based on these relationships, the system can generate many trajectories in a simulated experimental environment and obtain the corresponding simulated RSRP. Meanwhile, the team can verify the effect of the simulation method on the real dataset. For the interpolated real trajectory in the data set of Testing_B, the RSRP generated by the simulation algorithm has a similar distribution with the real RSRP, as shown in Fig. 30. b) Machine Learning (ML)-based Position Estimation: When there is enough data to form the training data set, CatBoost is used to complete the end-to-end position estimation task. CatBoost is a supervised learning algorithm based on gradient boosting and has excellent performance while reducing overfitting and the time spent on tuning. The CatBoost model takes the true coordinate [xt,y t]as a label and the RSRP and the differences of RSRPs received by two different pRRUs as the input vector Inputtwhich can be represented as shown in (7) Inputt=[Rt 0,...,R t N−1,DR t 0,1,...,DR t N−2,N−1](6) DRt i,j =Rt i−Rt j(7) where the Nmeans that there are NpRRUs while Rt i−1means that the RSRP received by the ith pRRUs at time t. 112 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 Fig. 30. Distribution of the real RSRP received by pRRU0 and the simulated RSRP of pRRU0. c) Position Estimation Correction: Using CatBoost, a position based on RSRP can be estimated, however, RSRP has a large random noise that makes the estimation unstable. To achieve a more stable estimation, the system uses additional information to optimize the position estimation. For example, the users cannot pass through obstacles (e.g., furniture) because of space constraints that can be inferred from the reachable area and the trajectory in the data set. Furthermore, due to time constraints, it is unlikely that the two adjacent estimated positions are far apart. Therefore, abnormal position estimations are fixed based on the reachable area, historical position information, and motion direction. d) Smoothing and Resampling Trajectory: With these data processing methods, reliable and stable position estimation can be reached. However, the normal trajectory should be smooth and continuous. Therefore, a KF is used to smooth the trajectory in real-time. Then, the relationship between the recent time and the estimated position is fit and resampled to obtain the results required by the competition. 2) Team TX8: In the IPIN 2022 competition track 8, two sets of data are given to calibrate the algorithmic models. In each dataset, the TOA and RSRP measurements are provided. As already identified by team mobile, high precision positioning requires estimating the TAEs accurately, i.e., to get the unknown timing errors among the receivers in TRPs. In addition, since the indoor environment is very complex, there exists a mixture of LOS paths, weak LOS paths, NLOS paths, causing a lot of outliers in the TOA and RSRP measurements. To achieve highprecision positioning, the outliers must be handled reasonably. FortheTAEsestimation,theRSRPmeasurementsareemployed. Generally, the RSRP can be expressed as follows: σ=A−10ηlog(d)+ε(1)(8) where σis the RSRP measurement, dis the distance between the terminal and TRP, Aand ηare the model parameters, which can be determined by using the datasets, and εis the measurement noise. In this manuscript, a Bayesian filter is employed to estimate the TAEs and the location of the terminal simultaneously. The state space model for the TAEs and location estimation can be written as [53] xk=Fk−1xk−1+Gk−1wk−1(9) zk=h(xk)+vk(10) Fk−1= ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 1T00000 0100000 001T000 0001000 0000100 0000010 0000001 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ (11) Gk−1= ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ T2 20 000 T0 000 0T2 2000 0T000 00100 00010 00001 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ (12) h(xk)=[h1(xk)···h3(xk)g0(xk)···g3(xk)] (13) hi(xk)=di,k −d0,k +bi, k (14) gi(xk)=Ai−10ηilog(di,k)(15) di,k =(xk−xTRPi)+(yk−yTRPi)+4 (16) where xk=[xkvx,k ykvy,k b1,k b2,k b3,k]T,xkand ykare the horizontal coordinates of the terminal at time k,vx,k, and vy,k are the horizontal velocities of the terminal at time k,b1,k, b2,k, and b3,k are the TAEs of the TRPs at time k,wk−1is the process noise at time k−1, Tis the sampling interval, zk=[γ1,k γ2,k γ3,k ρ0,k ρ1,k ρ2,k ρ3,k ],γi,k is the difference of the TOA measurements of TRPi and TRP0, ρi,k is the RSRP measurement of TRPi at time k,xTRPi and yTRPiare the horizontal coordinates of the TRPi, Aiand ηiare the model parameters of the TRPi, vkis the measurement noise at time k. Considering the outliers in the measurements, vkis modeled as a heavy-tailed non-Gaussian noise. Since the MCC-based EKF can handle the heavy-tailed non-Gaussian noise by using a robust cost function [54], the MCC-based EKF is utilized to estimate the TAEs and the location of the terminal based on the state space model proposed above. The workflow of the developed algorithm is shown in Fig. 31. 3) Team DYS-BUPT: In recent years, the DYS-BUPT team has been committed to 5G indoor positioning research to meet the positioning needs under different indoor scenes. In track 8, for the positioning scheme in the indoor office scene, DYSBUPT solution mainly includes three parts and the system block diagram is shown in Fig. 32. a) Neural network regression position: Inthedataenhancementpart,inorder to increasetheamountofdata formodel training and improve the generalization ability of the model, neural networks are used to fit the wireless channel propagation model, so as to systematically generate more training samples to expand the training dataset, as shown in Fig. 33. In the later position settlement part, considering that the data areRSRPcollectedalongthecontinuousmotion,thesinglepoint POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 113 Fig. 31. Workflow of the developed algorithm. Fig. 32. System block diagram of indoor positioning scheme. Fig. 33. Module A system block diagram. matching method may cause a large positioning error. The position of the dynamic target is constrained by space and time, so a recursive neural network is adopted. This network is no longer like the traditional fingerprint positioning, which only relies on the fingerprint of a certain point to locate each time. Instead, it takes into account the correlation of RSRP measurements in the continuous trajectory, considers the space and time constraints of the motion trajectory on the basis of single point matching, realizesthecorrelationoftimeandlocationinformationofRSRP Fig. 34. Module B system block diagram. in the trajectory, and transforms discrete positioning tasks into continuous time series feature discovery tasks. b) KF fusion position: In the part of data processing, the clock error between TRP and UE is calculated based on the first point given in the regression model and the TOA information collected and then obtain unbiasedTOA data based on the clock error and the originalTOA. Finally, according to 3·σThe criterion filters the data and smooths the abnormal value TOA, as shown in Fig. 34. In the data solution part, the position at the previous time is fused with the data collected at this time through Kalman filtering to obtain the Kalman location estimation at time k.At the same time, the system obtains the LLOP estimation value at time kby solving the LLOP algorithm, and combines the two positioning methods with empirical weighting to obtain the positioning estimation value of Module B. c) Numerical weighted fusion and trajectory correction: InmoduleC,thelocationestimatesobtainedfrommodules A and B are empirically weighted to achieve better results. Then, consider the actual situation, and calibrate the areas that cannot be reached by pedestrians, such as points outside the room, to obtain more reliable results. XI. RESULTS In this section, we report the overall scores for each track and its competitors in editions 2021 and 2022 of the IPIN Competition. Table VI presents the results for the 2021 edition, while Table VII presents the results for the 2022 edition. In both editions, as usual in the IPIN competition, the reported results correspond to the third quartile of the error metric, which is the 2-D positioning error plus a floor penalty of 15 m. Each track defined a cutoff threshold to be eligible for a prize. i.e., teams providing an error larger than the cutoff were not awarded a prize. Given the large errors provided by some teams in 2021, very large errors were reported differently in 2022. For the 2022 edition, the errors larger than three times the cutoff value are represented by >3×15 in tracks 3 and 4, and by >3×40 in track 6. 114 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 TABLE VI RESULTS 2021 TABLE VII RESULTS 2022 Table VIII gives the main techniques used in the systems described within this manuscript, where inertial techniques (PDR or PDR with ZUPT) are used by all teams in tracks 3, 4, and 6. Fingerprinting is also used by smartphone-based positioning, using only Wi-Fi, BLE, or the combination of both signals and magnetic field. Floor estimation exploits barometer information. Fig. 35. Trajectory plot of track 2 winner’s best scoring trial. The estimated trajectory (est trj) is evaluated against the ground truth trajectory (gt trj) at predefined reference points (ref pts) based on horizontal Euclidean distances to the corresponding estimated points (est pts) with an additional penalty for incorrectly estimated floor (marked with floor error). As far as algorithms are regarded, a PF is used in Tracks 3 and 7. It is combined with map information. However, some teams also used environmental information without combining it with PFs, see the description of teams X-LAB and WHU-GD. Another important element from the table is that some teams participated in more than one track: X-LAB and imec-WAVES, although they use different techniques in different Tracks. Thus, we see from the table that the tracks created correspond to different solutions and there is little overlap among different tracks. The most overlap is among tracks 3 and 4, however, they are physically very different: track 3 is based on smartphones thatcanexploitthecombinationofdataprovidedbylowaccurate built-in sensors of different nature; and track 4 is solely based on a higher-quality IMU, which enables the integration of better inertial information in the navigation algorithms. KF and its variants are used in tracks 3, 7, and 8. A. Track 2 Two teams participated in Track 2 competition: team1 was SZUSCRI from Shenzhen University and Smart City Research Institute; team2 was CamLoc from Beijing University of Posts and Telecommunications. During scoring trials, the competitors connected to the testing server and started receiving testing images and returning back pose estimations. Each subsequent image was sent only after receiving the pose estimation of the previous image. The competitors had no indication of which images were to be used as reference points. At the end of the scoring trial, the reference points were used tocalculatepositionerrorsasasumoftwo terms: a positionerror calculated from the Euclidean (horizontal) distance between the estimated position and the corresponding ground truth and a floor error penalty of 15 m. The third quartile error of the best scoring trial for each team was 3.2 m for team SZUSCRI and 2.1 m for team CamLoc (see Table VII Track 2: Camera). The trajectory of the scoring trial for the winner CamLoc is shown in Fig. 35. POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 115 TABLE VIII MAIN TECHNIQUES USED BY THE TEAMS PROVIDING FULL DESCRIPTION Fig. 36. Trajectory plot of track 3 (2021) best scoring trial (top) and a postprocessed trial (bottom). B. Track 3 In 2021, a total of 16 teams registered in the competition but only four of them were able to submit the results with the EvaalAPI. This means a significant drop in participation when compared to previous editions, where 11 (2020), 12 (2019), and 13 (2018) teams submitted the results. Moreover, the best team scored an error of 4.4 m while the runner-up’s error was 7.9 m, in phase with what is expected for smartphone-based solutions according to previous on-site competitions [5].The lower participation and the scores in 2021 reinforced the idea that the EvaalAPI was necessary to stress real-world on-site evaluation features in track 3. Fig. 36 shows the trajectory for the winner in 2021 (top plot) and the trajectory of a post-processed trajectory like in previous editions (bottom plot). Fig. 37. Trajectory plot of track 3 (2022) best scoring trial. In 2022, a total of ten teams registered in the competition but only seven of them were able to start the procedure to submit the results with the EvaalAPI. This means that interested teams made an effort to adopt the EvaalAPI for evaluation. The number ofteamsprovidingreliableresultswasinphasewiththeprevious edition, but only two teams repeated and participated again. In this case, the best team scored an error of 30.1 m while the runner-up’s error was 39.8 m, both (of which are) above the cutoff of 15 m of track 3. Therefore none of the participating teams was eligible for the award as the overall lowest error was below the expectations for a smartphone-based positioning solution. Fig. 37 shows the trajectory for the best-performing trial in 2022, where we can observe very large positioning errors in several parts of the trajectory. The results of the other three participating teams are not shown as the errors were three times larger than the cutoff of 15 m. In both editions, the use of advanced PF and/or KF is a core element to deal with smartphone data, including Wi-Fi fingerprinting and internal measurements. This requires properly representing the information contained in the provided floorplans. 3-D graphs are only used by one team, which seems a promising solution. In addition, the PF and KF filters need to have good 116 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 Fig. 38. Evolution of the two first scores of track 4 over the last five years. strategies for settling the initial position and orientation as well as correcting locations in case of severe deviation. In the 2022 edition, where the number of different devices used was higher and without BLE infrastructure supporting indoor localization, the competitors scored a positioning error much worse than usual. This even happened to the system developed by the imec-WAVES team, who has participated in the competition for many years. This highlights the relevance of having multiple heterogeneous environments to test every single solution, as a good promising indoor positioning system may not fit all environments. Analyzing the best performing trials (see Figs. 36 and 37), we can observe that the outputs provided by the competitors are more realistic than those provided in previous editions with off-line evaluation. First, trajectories are not shown as perfectly drawn straight lines as in previous editions (see [4] and Fig. 36) as noise from sensors is visible in the trajectories as zigzag movement, drifts or messy trajectories in a challenging walking style. While short-term displacements can be captured, see text IPIN between points 35–39 in 2021 and text T3 between points 58–59 in 2022, noise and drift remain there. Second, the trajectory cannot be fixed a posteriori, so a large error in the initial location can end up in a large positioning error over the whole trajectory as it happened in 2022. Third, integration with other sources to fix the location in real-time, like map-matching may be more challenging and filters like PF and KF are computationally demanding. Those three elements can be seen in the simulatedphonecallofaround1 minperformedin2021between points 25 and 26. The trajectory of the best trial is messy during the phone call, it is not fixed either in the short or in the long term, and it transverses some walls, while in a postprocessed trajectory, the trajectory between those two key points is drawn as a straight line. C. Track 4 In 2021, a total of three teams registered for track 4, of which only two were able to deliver results via the EvaalAPI platform. This year was the first time that the EvaalAPI platform was deployed for track 4. As shown in Fig. 38, there is a large gap in terms of final results before and after the introduction of the EvaalAPI platform. The best score before introducing the EvaalAPI platform, where full CSV files were Fig. 39. Trajectory plot of track 4 (2022) best scoring trial. shared with competitors in a postprocessing mode, was 0.5m. However, it increased to 62 m in 2021 with EvaalAPI in a quasi-real-time mode. Although this can be partly explained by competitors’ lack of familiarity with the new platform, the main reason is the causal effect, which does not provide access to future information and makes forward-backward adjustment impossible. Compared with other tracks, track 4 is particularly affectedbythiseffectduetotheerroraccumulationoftheinertial sensors. Not many absolute “resets” can be performed on track 4, especially in GNSS-denied environments. In 2022, a total of five teams chose to compete in track 4. Four teams were able to output quasi-real-time results via the EvaalAPI. We note that this is better than the previous edition, both in terms of the number of registered teams and teams able to produce results. As we can see in Fig. 38, the winner achieves an accuracy of about 77 m. Even if it seems worse than the previous editions, a real improvement was achieved taking into account the complexity of the trajectory this year as displayed in Fig. 39. We can see clearly that compared with the ground truth pattern (in green), the competitor’s estimated trajectory suffers from a continuous rotation drift, which is typical for the dead reckoning algorithm. The figure shows the best 2-D trace since the introduction of EvaalAPI. Its final score of 76.9mversus 61.9 m for the winner of 2021 is explained by the poor quality of floor estimation that led to 15 m of penalties. D. Track 7 In 2021, a total of seven teams provided results for the challenge (see Table IX). Different approaches were investigated by the competitors: three teams relied on LOS-error mitigation approaches, three relied on PFs, and one on a C-SLAM approach. Due to the environment changes, the PF-based approaches deteriorate heavily in performance from Test 1 to 2, as the models are fitted to the specific environments. The EMI-based approaches do not exhibit this problem, as the environmental conditions stay similar.ThesameholdsfortheC-SLAMapproach.TeamISCAS and Waves shared the first place with almost identical 75th error POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 123 Fang Zhao (Member, IEEE) received the B.S. degree in computers and applications from the School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China, in 1990, and the M.S. and Ph.D. degrees in computer science and technology from the Beijing University of Posts and Telecommunications, Beijing, China, in 2004 and 2009, respectively. She is currently a Professor with the School of Software Engineering, Beijing University of Posts and Telecommunication. Her research interests include mobile computing, location-based services, and computer networks. Yue Zhuge received the B.S. degree in computer science and technology from the Wuhan University of Technology, Wuhan, China, in 2021. She is currently working toward the M.S. degree in computer application technology with the University of Chinese Academy of Sciences, Beijing, China. Her research interests include computer vision, simultaneous localization, and mapping. Haiyong Luo (Member, IEEE) received the B.S. degree in information engineering from the Department of Electronics and Information Engineering, Huazhong University of Science and Technology, Wuhan, China, in 1989, the M.S. degree in communication and information systems from the School of Information and Communication Engineering, Beijing University of Posts and Telecommunication, Beijing, China, in 2002, and the Ph.D. degree in computer science from the University of Chinese Academy of Sciences, Beijing, in 2008. He is currently an Associate Professor with the Institute of Computer Technology, Chinese Academy of Science, Beijing. His main research interests are location-based services, pervasive computing, mobile computing, and Internet of Things. Antoni Perez-Navarro (Member, IEEE) received the bachelor’s and Ph.D. degrees in physics from the Universitat Autónoma de Barcelona, Bellaterra, Spain, in 1995 and 2000, respectively. Between 2017 and 2020, he held the position of Deputy Director of Research with the eLearn Center, Universitat Oberta de Catalunya (UOC) and is Lecturer with the Computer Science, Multimedia, and Telecommunication Department (EIMT Department), since 2005. He is also a Member eHealthLab research group. He is currently the Director of the Technological Observatory with the EIMT department. Apart from his activities at UOC, he works, since the year 2007 with Escola Universitària Salesiana de Sarrià (EUSS). His teaching activities range from the fields of physics and GIS in telecommunication engineering, computer science, multimedia, and industrial engineering. He has authored or coauthored several papers in international journals in all these topics and acts as a reviewer of several journals. His main research interests are indoor positioning, prevention of diseases via smartphones, and e-Learning. Dr. Perez-Navarro is part of the Technical Program Committee of IPIN and is one of the Chairs of IPIN 2021. Antonio Ramón Jiménez was born in Santander, Spain, in 1968. He received the degree in physics and computer science and the Ph.D. degree in physics from the Universidad Complutense de Madrid, Madrid, Spain, in 1991 and 1998, respectively. Since 1993, he has been with the Center de Automation y Robotics, Spanish Council for Scientific Research, Madrid, where he holds a research position. He has authored more than 100 articles in journals and conference proceedings. His current research interests include local positioning solutions for indoor/ global positioning system-denied localization and navigation of persons and robots, signal processing, Bayesian estimation, and inertial-ultrasonic-RFID sensor fusion. Dr. Ruiz is a Reviewer for many international journals and projects in the field. Han Wang (Member, IEEE) received the B.E. and Ph.D. degrees in electrical and electronics engineering from Nanyang Technological University, Singapore, in 2016 and 2021. Since 2021, he is working as a Research Assistant with the Huawei Technologies, Company Ltd., Shen Zhen, China, where he is currently a Researcher. His research interest, include simultaneous localization and mapping, pedestrian dead reckoning, and computer vision. Hengyi Liang (Member, IEEE) received the B.S. degree in microelectronics from the University of Electronic Science and Technology of China, Chengdu, China, in 2016, and the Ph.D. degree in computer engineering from Northwestern University, Evanston, IL, USA, in 2021. He is currently a Researcher with Huawei Technologies, Company, Ltd. His research interests include indoor positioning, cyber-physical systems, and connected and autonomous vehicles. Cedric De Cock received the M.S. degree in electronics and ICT Engineering Technology from Ghent University, Ghent, Belgium, in 2020. In 2020, he became a Member of the imec-WAVES Group, Department of Information Technology, Ghent University. His research interests include IMU-enabled indoor positioning and Bayesian filtering algorithms. David Plets (Member, IEEE) has been a Member of the imec-WAVES Group, Department of Information Technology, Ghent University, Ghent, Belgium, since 2006. He is currently an Associate Professor with the Ghent University. His current research interests include localization techniques and the IoT, for both industryand health-related applications, and also involved in the optimization of wireless communication and broadcast networks. 124 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 Yan Cui received the B.S. degree in 2021 from the Nanjing University of Aeronautics and Astronautics, Nanjing, China, where he is currently working toward the master’s degree with the Navigation Research Center. His research interests include indoor inertial navigation and indoor multisource navigation. Zhi Xiong received the M.S. and Ph.D. degrees from the Nanjing University of Aeronautics and Astronautics, Nanjing, China, in 2001 and 2004, respectively. He joined NUAA, where he has been a Full Professor with the College of Automation, since 2011. In 2013, he was an Academic Visiting Fellow with the University of Southern California, USA. He has ten years’ experience in the inertial navigation field and has led more than 30 navigation system development projects. His main research interests include inertial navigation, small aircraft navigation, brain-like navigation, and multisource fusion. Xiaodong Li received the B.S. degree in automation in 2020 from the Nanjing University of Aeronautics and Astronautics, China, where he is currently working toward the Ph.D. degree with the Department of Automation Engineering. His research interests include indoor inertial navigation and multisource fusion. Yiming Ding received the M.S. degree from Soochow University, Suzhou, China, in 2018. He is currently working toward the Ph.D. degree with the Navigation Research Center, Nanjing University of Aeronautics and Astronautics, Nanjing, China. His research interests include pedestrian dead reckoning and wearable sensors. Fernando Javier Álvarez Franco (Senior Member, IEEE) received the M.Sc. degree in physics from the University of Sevilla, Sevilla, Spain, in 1998, the Ph.D. degree in electronics from the University of Alcalá, Alcalá de Henares, Spain, in 2006, the M.Sc. degree in electronic engineering from the University of Extremadura, Badajoz, Spain, in 2012, and the M.Sc. degree in signal theory and communications from the University of Vigo, Vigo, Spain, in 2014. Since 2001, he has been with the Department of Electrical Engineering, Electronics and Automation, University of Extremadura, where he is currently a Full Professor and the Head of the Sensory Systems Research Group. In 2008, he joined the Intelligent Sensors Laboratory, Yale University, New Haven, CT, USA, as a Postdoctoral “Jose Castillejo” Fellow. His current research interests include local positioning systems, acoustic signal processing, and embedded computing. Fernando Jesús Aranda Polo (Student Member, IEEE) received the B.S. degree in physics from the University of Extremadura, Badajoz, Spain, in 2018, the M.D. degree with a specialization in the simulation of physical phenomena as part of the master’s degree in simulation of science and engineering problems with the University of Extremadura, respectively. He is currently working toward the Ph.D. degree with Sensory System Research Group. Since 2019, he has been a Member of the Sensory System Research Group. His research interests include fingerprinting positioning, machine learning, and radio frequency signal modeling. Felipe Parralejo Rodríguez (Student Member, IEEE) received the B.S. degree in physics and M.D. degree in engineering and science simulation from the Universidad de Extremadura, Badajoz, Spain, in 2019 and 2020, respectively. He is currently working toward the Ph.D. degree with Sensory Systems Research Group, Department of Electrical Engineering and Electronics. His current research interests include mmWave radar-based human monitoring and the application of machine learning for the processing of the information obtained from intelligent sensors. Adriano Moreira (Senior Member, IEEE) received the degree in electronics and telecommunications engineering and the Ph.D. degree in electrical engineering from the University of Aveiro, Aveiro, Portugal, in 1989 and 1997, respectively. He is currently an Associate Professor of habilitation with the School of Engineering, University of Minho, Braga, Portugal and a Researcher with Algoritmi Research Centre, Guimãres, Portugal. He is the Director of the MAP-tele Doctoral Program in telecommunications. He has authored more than 100 scientific publications in conferences and journals, and the author of one patent in the area of computational geometry. In the past few years he participated in many research projects funded by national and European programs. His research interests include urban computing, human mobility analysis, indoor positioning and simulation of wireless, mobile networks in urban contexts, and the creation of technologies for smart places. Dr. Moreira was the recipient of the first prize on the off-site track of the EvAAL-ETRI Indoor Localization Competition (IPIN 2015 and 2017). He is currently the Chair of the Steering Committee of the International Conference on Indoor Positioning and Indoor Navigation and a Member of the ICL-GNSS conference Steering Committee. POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 125 Cristiano Pendão received the master’s degree in telecommunications and informatics from the University of Minho, Braga, Portugal, and the Ph.D. degree in telecommunications/ computer science engineering from the University of Minho, University of Aveiro, Aveiro, Portugal, and University of Porto, Porto, Portugal, respectively. He is currently a Professor with the Department of Engineering, School of Sciences and Technology, University of Trás-os-Montes and Alto Douro, Vila Real, Portugal. He was a Research Member with ALGORITMI Research Centre, University of Minho. He has coauthored numerous publications in international conferences and journals and has actively participated in several scientific, innovation, and commercial projects within the industry. His research interests include mobile computing, positioning and navigation, computer vision/ perception, and machine learning. Ivo Silva received the M.Sc. degree in telecommunications and informatics engineering and the Ph.D. degree (MAP-tele Doctoral Program) in telecommunications from the University of Minho, Minho, Portugal, University of Aveiro, Aveiro, Portugal, and University of Porto, Porto, Portugal, in 2016 and 2022, respectively. He is a Researcher with Algoritmi Research Centre and an Invited Professor with the University of Minho. He has coauthored several scientific papers in conferences and journals, and has participated in R&D projects in partnership between academia and industry. His research interests include Industry 4.0, indoor positioning and navigation, vehicle localization, mobile computing, and smart devices. Miguel Ortiz received the M.Sc. degree in mechanics, automation, and engineering from the National School of Arts and Crafts, Meknes, Morocco, in 2001. He is currently a Research Engineer with GEOLOC Laboratory, University Gustave Eiffel, Bouguenais, France. He joined the lab after six years spent in a company where he managed systems architecture for automotive applications. He is currently an expert in embedded electronic systems. Since 2017, he has been the Convenor of CEN/CENELEC TC5-WG1 named “Navigation and positioning receivers for road applications.” Since 2022, he has been the Convenor of ISO-TC20-SC14-WG8 named “’Downstream space services and space-based applications.” He has 15 years of experience in the GNSS domain (research/engineering/standardization). Since 2019, he has been the Deputy Head of GEOLOC Laboratory, Gustave Eiffel University, Bouguenais, France. His research interests include software and hardware developments for both ITS (Intelligent Transport Systems) and pedestrian navigation research field. Ni Zhu (Member, IEEE) received the Engineering degree in aeronautic telecommunications from the National School of Civil Aviation, Toulouse, France, in 2015, and the Ph.D. degree in science of information and communication from the University of Lille, Lille, France, in 2018. She is currently a Research Fellow with the Laboratory GEOLOC, University Gustave Eiffel, Bouguenais, France. Her research interests include specialization in GNSS channel propagation modeling in urban environments, positioning integrity monitoring for terrestrial safety-critical applications, and multisensory fusion techniques for indoor/outdoor pedestrian positioning assisted by artificial intelligence. Since 2020, she has been the Co-Chair of the foot-mounted IMUbased positioning track of indoor positioning and indoor navigation competition. Ziyou Li received the M.Sc. degree in aerospace navigation and telecommunication from French Civil Aviation University (Ecole Nationale d’Aviation Civile), Toulouse, France, in 2021. He is currently working toward the Ph.D. degree in GNSS pedestrian navigation and integrity monitoring in challenging environments with GEOLOC Laboratory, University Gustave Eiffel, Bouguenais, France. He joined the GEOLOC Laboratory in 2021. His research interests include GNSS pedestrian navigation and integrity monitoring in challenging (deep urban and light indoor) environments. Valérie Renaudin (Member, IEEE) received the M.Sc. degree in geomatics engineering and the Ph.D. degree in computer, communication, and information sciences from the Swiss Federal Institute of Technology Lausanne, Lausanne, Switzerland, in 1999 and 2009, respectively. She is currently a Professor with Gustave Eiffel University, Bouguenais, France. She was the Technical Director with SWISSAT, Schwyz, Switzerland, where she developed real-time positioning solutions based on a permanent global network of satellite navigation systems (GNSS), and a Senior Research Associate with the University of Calgary, Calgary, AB, Canada. She currently heads the Geopositioning Laboratory, Gustave Eiffel University, where she has built a team specializing in the positioning and navigation of travelers. Her research interests include indoor/outdoor navigation methods and systems using GNSS, as well as inertial and magnetic data, especially for pedestrians to improve sustainable personal mobility. Dr. Ranaudin was the recipient of several awards, including a Marie Curie European Grant for smartWALK project. She founded NAV4YOU, Bouguenais, France, in 2021, developing location-based services for the safety of firefighters in intervention, defence, and underground activities. She is the Editor-in-Chief of the new open access IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION that she launched in 2022. She is also a member of the steering committee of the international conference “Indoor Positioning and Indoor Navigation.” Dongyan Wei received the B.S. degree in communication engineering from the University of Electronic Science and Technology of China, Beijing, China, in 2006, and the Ph.D. degree in signal and information processing from the Beijing University of Post and Telecommunication, Beijing, in 2011. He is currently a Research Fellow with Aerospace Information Research Institute, Chinese Academy of Science, Beijing, China. He has authored one book, more than 30 articles, and more than 20 inventions. His research interests include indoor position, multisensor fusion and positing in wireless network. Dr. Wei is the TPC Member of IPIN 2019 and the Deputy Chair of IPIN 2022. Xinchun Ji received the B.S. and M.S. degrees in guidance navigation and control from the Beijing University of Aeronautics and Astronautics, Beijing, China, in 2010 and 2013, respectively. He is currently working toward the Ph.D. degree in electronic information with Northwestern Polytechnical University, Xi’an, China. He is currently an Engineer with Aero Information Research Institute, Chinese Academy of Science, Beijing. His research interests include multisensor fusion and geomagnetic matching. 126 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 Wenchao Zhang received the B.S. degree in surveying engineering from the China University of Mining and Technology, Xuzhou, China, in 2013, the M.S. degree in surveying engineering from Information Engineering University, Zhengzhou, in 2016, and the Ph.D. degree in signal and information processing from the University of Chinese Academy of Sciences, Beijing, in 2020. He is currently an Assistant Researcher with Aero Information Research Institute, Chinese Academy of Science (CAS). His research interests include multiinformation fusion method, integrated navigation algorithm, and pedestrian autonomous positioning algorithm. Yan Wang received the B.Eng. degree in chemical engineering and technology and the M.S. degree in computer applied technology from the China University of Mining and Technology, Xuzhou, China, in 2016 and 2019, respectively. He is currently working toward the Ph.D. degree with GNSS Research Center, Wuhan University, Wuhan, China. His research interests include indoor navigation, sensor fusion algorithm, and computer vision. Longyang Ding received the B.Eng. degree (Hons.) in surveying and mapping engineering in 2022 from Wuhan University, Wuhan, China, where he is currently working toward the M.Eng. degree in navigation, guidance, and control with GNSS Research Center. His research interests include GNSS/INS integration for land vehicle navigation and mobile robot state estimation. Jian Kuang received the B.Eng. and Ph.D. degrees in geodesy and survey engineering from Wuhan University, Wuhan, China, in 2013 and 2019, respectively. He is currently a Postdoctoral Fellow with GNSS Research Center, Wuhan University, Wuhan, China. His research interests include inertial navigation, pedestrian navigation, and indoor positioning. Xiaobing Zhang received the B.S. degree in computer science and technology from Peking University, Beijing, China, in 2003. He is currently a Senior Algorithm Expert with Autonavi Software Company Ltd., Beijing, China. His current research interests include inertial navigation with smartphones and crowdsourcing positioning in GNSS-denied scenarios. Zhi Dou received the B.Eng. degree in electronic information engineering from the Hebei University of Technology, Tianjin, China, in 2014, the Ph.D. degree with the School of Electronic Information Engineering, Tianjin University, China, in 2016. His current research interests include machine learning and vehicle navigation and positioning with smartphones. Chaoqun Yang received the B.Eng. degree in electronic engineering in 2010 from Tsinghua University, Beijing, China, where he continued to study for the Ph.D. degree in electronic engineering and dropped out in 2018. His research interests include machine learning and GNSS/inertial fusion. Sebastian Kram received the M.Sc. degree in electrical and communication engineering with FAU Erlangen-Nürnberg, Erlangen, Germany, in 2017. He then joined the Locating and Communication Systems Department, Fraunhofer IIS, Erlangen, Germany. Since 2020, he has been with the Navigation Group, Chair for Information Technology (Communication Electronics) University of Erlangen-Nuremberg, Erlangen, Germany. His research focuses on radio signalbased positioning using both modeland data-driven methods. Maximilian Stahlke received the master’s degree in electronic and mechatronic systems from the Institute of Technology Georg Simon Ohm, Nuremberg, Germany, in 2020. Since 2020, he has been with the Hybrid Positioning and Information Fusion groupworks, Precise Positioning and Analytics Department, Fraunhofer IIS, Erlangen, Germany. His research interests include hybrid positioning for radio-based localization systems with the focus on modeland data-driven information fusion. Christopher Mutschler received the diploma and Ph.D. degrees in computer science from Friedrich-Alexander-University ErlangenNuremberg (FAU), Erlangen, Germany, in 2010 and 2014, respectively. From 2017 to 2019, he was Head and the Chief Scientist with the Machine Learning and Information Fusion Group. He is currently the Head of the Precise Positioning and Analytics Department with Fraunhofer IIS, Nuremberg, Germany. He simultaneously is part-time Scientific Staff with the FAU, offering courses on machine learning. His research interests include machine learning and hybrid sensor fusion for radio-based locating systems. Sander Coene received the joint M.Sc. degree in nuclear fusion science and engineering physics from the Universität Stuttgart, Stuttgart, Germany, the Université de Lorraine, Lorraine, France, Universiteit Ghent, Ghent, Belgium, Universidad Complutense de Madrid, Madrid, Spain, and Universidad Carlos III de Madrid, Madrid, in 2016. He is currently working toward the Ph.D. degree in electrical engineering with the Ghent University, Ghent. After a few years of research in the space industry, he joined the WAVES group of the Department of Information Technology, Ghent University in 2020. His research interests include ultrawideband ranging and localization algorithms for indoor applications. POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 127 Chenglong Li (Member, IEEE) was born in Nanchong, China, in May 1994. He received the Ph.D. degree in electrical engineering from Ghent University, Ghent, Belgium, in 2022. From 2018 to 2022, he was a Research Assistant with the Department of Information Technology, Ghent University-imec. He is currently a Lecturer with the College of Electronic Science and Technology, National University of Defense Technology, Changsha, China. His current research interests include positioning and navigation, wireless sensing, wireless channel modeling, and mobile computing. Dr. Li was the recipient of the International Union of Radio Science Young Scientist Award and European Microwave Association Student Grant in 2022. He was the shared winner of IPIN Competition 2021 track 7. Alexander Venus (Student Member, IEEE) received the B.Sc. and Dipl.-Ing. (M.Sc.)degrees (with highest honors) in biomedical engineering and information and communication engineering in 2012 and 2015, respectively, from the Graz University of Technology, where he is currently working toward the Ph.D. degree. From 2014 to 2019, he was a Research and Development Engineer with Anton Paar GmbH, Graz. He is currently a Project Assistant with the Graz University of Technology. His research interests include radio-based localization and navigation, stochastic modeling, inference on graphs, and estimation/detection theory. Erik Leitinger (Member, IEEE) received the Dipl.-Ing. (M.Sc.) and Ph.D. degrees (with highest honors) in electrical engineering from Graz University of Technology, Graz, Austria in 2012 and 2016, respectively. From 2016 to 2018, he was a Postdoctoral Researcher with the Department of Electrical and Information Technology, Lund University, Lund, Sweden. He is currently a University Assistant with Graz University of Technology. His research interests include inference on graphs, localization and navigation, multiagent systems, stochastic modeling and estimation of radio channels, and estimation/detection theory. Dr. Leitinger was the Co-chair of the special session “Synergistic Radar Signal Processing and Tracking” at the IEEE Radar Conference in 2021. He is Co-organizer of the special issue “Graph-Based Localization and Tracking” in the Journal of Advances in Information Fusion.Hewas the recipient of the an Award of Excellence from the Federal Ministry of Science, Research and Economy for his Ph.D. Thesis. He is an Erwin Schrödinger Fellow. Stefan Tertinek received the Dipl.-Ing. degree in electrical engineering from the Graz University of Technology, Graz, Austria, in 2007, and the Ph.D. degree in electrical engineering from University College Dublin, Dublin, Ireland, in 2011. From 2011 to 2018, he was with Danube Mobile Communications Engineering GmbH and Co KG (majority owned by Intel Austria GmbH), Linz, Austria, as a RF System Engineer involved in research and product development of multiple generations of cellular RF transceiver and modem platforms. In 2018, he joined NXP Semiconductors Austria GmbH and Co KG as a RF System Architect in the Product Line Secure Car Access, where he works on ultrawideband and bluetooth radio technologies. His research interests include localization, radar, and machine learning. Klaus Witrisal (Member, IEEE) received the Ph.D. degree (cum laude) from the Delft University of Technology, Delft, The Netherlands, in 2002, and the Habilitation from the Graz University of Technology, Graz, Austria, in 2009. He is currently an Associate Professor with the Signal Processing and Speech Communication Laboratory, Graz University of Technology, and Head of the Christian Doppler Laboratory for Location-aware Electronic Systems. His research interests include signal processing for wireless communications, propagation channel modeling, and positioning. Dr. Witrisal was an Associate Editor for IEEE COMMUNICATIONS LETTERS. He was the Co-Chair of the TWG “Indoor” of the COST Action IC1004, EWG “Localization and Tracking” of the COST Action CA15104. He was leading Chair of the IEEE Workshop on Advances in Network Localization and Navigation, and TPC (Co)-Chair of the Workshop on Positioning, Navigation and Communication. Yi Wang (Senior Member, IEEE) received the M.Sc. and Ph.D. degrees in the information engineering from the Beijing University of Posts and Telecommunications, Beijing, China, in 1997 and 2000, respectively. He joined Huawei research in 2005 working with 4G and 5G technologies. Since 2017, he has been working on 5G positioning and sensing. He owns 278 published patents and 77 scientific papers. Most patents are declared as ETSI type. Dr. Wang was the Chair of 5G millimeter-wave group in IMT-2020 promotion group in China during 2013–2018. Shaobo Wang received the graduation degree from Zhejiang University, Hangzhou, China, in 2000. From 2000 to 2008, he was responsible for the design of baseband receiver algorithms for the 3G OM’S system. From 2008, he led the R&D work of physical-layer and low-MAC algorithms in GSM, UMTS, LTE, and 5G NR systems, laying a foundation for the competitiveness of Huawei’s network products. Since 2017, he has been working as the Chief Engineer leading 5G Advanced Technology Research. He is a Senior Research Expert in the field of wireless communication air interface system and the Deputy Minister of RAN Research Department, Wireless Network, Huawei. Dr. Wang is/was the General and/or Program Co-Chair of many international conferences/workshops. Beihong Jin received the B.S. degree in computer science from Tsinghua University, Tsinghua, China, in 1989, and the M.S. and Ph.D. degrees in computer science from the Institute of Software, Chinese Academy of Sciences, Beijing, China, in 1992 and 1999, respectively. Since 1992, she has been with the Institute of Software, Chinese Academy of Sciences. She is currently a Full Professor with the Institute of Software, Chinese Academy of Sciences. Her research interests include mobile and pervasive computing, middleware, and distributed systems. 128 IEEE JOURNAL OF INDOOR AND SEAMLESS POSITIONING AND NAVIGATION, VOL. 2, 2024 Fusang Zhang received the M.S. and Ph.D. degrees in computer science from the Institute of Software, Chinese Academy of Sciences, Beijing, China, in 2013 and 2017, respectively. He is currently an Associate Professor with the Institute of Software, Chinese Academy of Sciences. His current research interests include mobile and pervasive computing, ad hoc network, and wireless contactless sensing. Chang Su (Graduate Student Member, IEEE) received the B.S. degree in software engineering from Xiamen University, Xiamen, China. He is currently working toward the M.S. degree in software engineering with the Institute of Software Chinese Academy of Sciences, Beijing, China. Zhi Wang received the B.S. degree in software engineering from Beijing Jiaotong University, Beijing, China. He is currently working toward the Ph.D degree in software engineering with the Institute of Software Chinese Academy of Sciences, Beijing, China. Siheng Li (Student Member, IEEE) received the B.S. degree in software engineering from Xiamen University, Xiamen, China. He is currently working toward the Ph.D. degree in theory and technology of network distributed computing with the Institute of Software, Chinese Academy of Sciences, Beijing, China. Xiaodong Li was born in Henan, China, in 1991. He received the B.S., M.S., and Ph.D. degrees in information and communication engineering from the Harbin Institute of Technology, Harbin, China, in 2014, 2016, and 2020, respectively. He is currently a Researcher with Purple Mountain Laboratories, Nanjing, China. His current research interests include indoor navigation and positioning, statistical signal processing, multisensor data fusion, and machine learning. Shitao Li is currently working toward the master’s degree with Southeast University, Nanjing, China. Mengguan Pan (Member, IEEE) received the B.S. degree in electronics information engineering and Ph.D. degree in signal and information processing from the School of Electronic Engineering, Xidian University, Xi’an, China, in 2013 and 2018, respectively. He is currently a Researcher with Pervasive Communication Research Center, Purple Mountain Laboratories, Nanjing, China. His research interests include parameter estimation in radar and wireless communication systems, array signal processing, wireless localization, and integrated sensing and communications. Wang Zheng received the Ph.D. degree in communication and information systemsfrom the College of Electronic and Information Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, China, in 2021. He is currently with Pervasive Communication Research Center, Purple Mountain Laboratories, Nanjing. His current research interests include angle of arrival estimation, indoor positioning, and sparse array signal processing. Kai Luo received the B.S. degree in electronic engineering and technology from the North China University of Technology, Beijing, China, in 2019. He is currently working toward the Ph.D. degree in electronic science and technology with the Beijing University of Posts and Telecommunications, Beijing. His current research areas include indoor and outdoor high-precision positioning and navigation, multisource fusion, signal systems, and signal processing. Ziyao Ma received the B.S. degree in communication engineering from the Beijing University of Posts and Telecommunications, Bejing, China, where he is currently working toward the Ph.D. degree in electronic science and technology. His current research interests include communication and navigation integration, 5G positioning, and 5G signal processing. Yanbiao Gao received the B.S. degree in electronic engineering from Northeastern University, Shenyang, China. He is currently working toward the Ph.D degree in electronic science and technology with the Beijing University of Posts and Telecommunications, Beijing, China. His current research include PRS positioning and 5G communications. POTORTÌ et al.: OFFSITE EVALUATION OF LOCALIZATION SYSTEMS: CRITERIA, SYSTEMS, AND RESULTS 129 Jiaxing Chang received the B.S. degree in electronic information science and technology from the Beijing University of Posts and Telecommunications, Beijing, China, where he is currently working toward the master’s degree in electronic science and technology. His current research interests include indoor localization and transfer learning. Hailong Ren received the B.S. degree in mechanical engineering in 2021 from the Beijing University of Posts and Telecommunications, Beijing, China, where he is currently working toward the M.S. degree in electronic information engineering. His current research interest includes 5G positioning and 5G signal processing. Wenfang Guo received the B.S. degree in electronic and information engineering from the University of South China, Hengyang, China. She is currently working toward the M.S. degree with the Beijing University of Posts and Telecommunications, Beijing, China. Her current research interests include the areas of wireless positioning and protocol for 5G NR positioning. Joaquín Torres-Sospedra received the Ph.D degree in computer science from Universitat Jaume I, Castelló, Spain, in 2011. He is now a Senior Researcher with the University of Minho (Guimarães, Portugal), where he works on Indoor positioning and machine learning for industrial applications. He has authored more than 170 articles in journals and conferences, and supervised 16 Master and 6 Ph.D. students. Dr. Torres-Sospedra is the Chair of the IPIN International Standards Committee and IPIN Smartphone-based offsite Competition. Open Access funding provided by ‘Consiglio Nazionale delle Ricerche-CARI-CARE-ITALY’ within the CRUI CARE Agreement