Performance Evaluation of Reinforcement Learning Algorithms for Learning Navigation in Unstructured Environments

Performance Evaluation of Reinforcement Learning Algorithms for Learning Navigation in Unstructured Environments Francesco Cancelliere, Giuseppe Sutera, Simone Palazzo, Dario Guastella, Concetto Spampinato, Giovanni Muscato DIEEI,Universit` a degli Studi di Catania, Catania, Italy {francesco.cancelliere@phd, giuseppe.sutera, simone.palazzo, dario.guastella, giovanni.muscato}@unict.it Fig. 1. MIDGARD Simulator multi-agent training in Forest Environment. Abstract—Autonomous navigation in unstructured outdoor environments presents significant challenges. While deep Reinforcement Learning (RL) has shown promise, it often faces difficulties in generalization and data efficiency. To address these issues, we evaluate the performance of four deep RL algorithms (PPO, A2C, SAC, and TD3) for a point-goal navigation task within MIDGARD, our photorealistic simulator built on Unreal Engine. By comparing success rates and reward progression, we demonstrate that PPO significantly outperforms the other algorithms. Our results indicate that on-policy methods are better suited for this navigation task, whereas off-policy methods struggle due to inefficient exploration and policy updates. Keywords: unmanned ground vehicle, reinforcement learning, autonomous navigation, outdoor unstructured environments. INTRODUCTION Navigating autonomous ground robots in complex outdoor environments is a persistent challenge. Classical methods often rely on rigid, hard-set rules and drift-prone sensors, limiting their adaptability in complex and diverse terrains, as explained in (1). For instance, traditional path planners require accurate, pre-built maps and can easily fail when encountering F. Cancelliere, G. Sutera and G. Muscato acknowledge financial support from PNRR MUR project PE0000013-FAIR. D.C. Guastella acknowledges support by the project PON R&I REACT-EU. All the authors are with Dept. of Electrical Electronic and Computer Engineering, University of Catania. Corresponding author: [email protected] unmapped obstacles or sensor noise. Their performance is fundamentally brittle in the very dynamic and unpredictable conditions that characterize unstructured terrain. Learningbased approaches, especially Deep Reinforcement Learning (DRL), offer a promising alternative by enabling agents to learn from interaction, similar to humans. By learning an endto-end mapping from sensory input to action, a DRL agent can develop more adaptive and resilient behaviors without depending on a perfect world model. However, RL learning still suffers from sample inefficiency and poor generalization, exacerbated by the sim-to-real gap, as shown in (2). To mitigate these issues, training in photorealistic simulators is crucial. This paper evaluates the performance of several well-established RL algorithms for an expert agent with access to privileged world information (e.g., precise obstacle locations). This expert policy is intended to later guide a sensor-based agent via imitation learning. We conduct our experiments in MIDGARD (3), our custom-built simulator, to identify the most effective algorithm for this navigation task. METHODOLOGY We frame the point-goal navigation task as a Markov Decision Process (MDP), defined by the tuple M= (S, A, P, R, γ). The agent must navigate to a target point while avoiding obstacles. State Space (S): The state includes the agent’s distance and angle relative to the target, along with the distance and angle to the three nearest obstacles. This assumes access to privileged information about the environment, which is readily available in simulation. Action Space (A): The action space is continuous, consisting of normalized linear velocity v∈[0,1] and angular velocity ω∈[−1,1], consistent with a non-holonomic skid-steer kinematic model. Reward Function (R): To guide the agent effectively, we designed a dense reward function: R=     +5,Goal Reached −2,Collision or Out-of-bounds 0.1 (dt−dt−1)−0.01 Otherwise This function rewards reaching the goal, penalizes failure, encourages reducing distance to the target (dt< dt−1), and discourages inefficient paths via a small time penalty. 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629854 215 Policy Network: The policy is represented by a fully connected neural network. The input layer receives the state information (distance/angle to target and 3 nearest obstacles), which is processed through two hidden layers of 64 nodes each. The output layer produces the continuous values for linear and angular velocity (Fig. 2). EXPERIMENTAL RESULTS We evaluated four RL algorithms from the stable-baselines3 libary (4): the on-policy methods PPO and A2C, and the off-policy methods SAC and TD3. All experiments were conducted with 200 concurrent agents, with each episode lasting a maximum of 2000 steps. The results, shown in Fig. 3 and Fig. 4, demonstrate that PPO is the most effective algorithm for this task. It achieved a stable success rate of approximately 65%, likely due to its ability to balance exploration and policy stability. A2C also performed reasonably well, reaching a 30% success rate, but its learning was slower and less stable. In contrast, the off-policy algorithms SAC and TD3 struggled significantly, plateauing at low success rates of 20% and 12%, respectively. Their poor performance can be attributed to inefficient exploration and the challenges of learning from nonstationary data stored in a replay buffer, which is less suited for this navigation task. The average reward curves closely mirrored the success rates, with PPO and A2C showing clear positive trends after an initial phase of negative rewards, while SAC and TD3 failed to escape suboptimal behaviors. CONCLUSIONS AND FURTHER WORK In this paper, we evaluated four prominent RL algorithms for autonomous navigation in our photorealistic simulator, MIDGARD. Our findings clearly indicate that PPO is the most effective algorithm for training an expert navigation policy with privileged information, outperforming A2C, SAC, and TD3. While PPO’s performance is strong, it could likely be improved with further hyperparameter tuning and reward shaping. The trained expert policy provides a solid baseline for future work. The next step is to use this policy to distill a more complex, sensor-driven policy via imitation learning, enabling the agent to navigate using high-dimensional inputs like RGB images or LiDAR scans, thereby bridging the gap to real-world deployment. Fig. 2. The policy network architecture. Fig. 3. Success Rate per Episode. Fig. 4. Average Sum of Undiscounted Reward per Episode. ACKNOWLEDGMENTS The authors acknowledge financial support from PNRR MUR project PE0000013-FAIR. The work of S. Palazzo was supported by ”PIAno di inCEntivi per la Ricerca di Ateneo 2024/2026”, University of Catania. REFERENCES [1] X. Xiao, B. Liu, G. Warnell, and P. Stone, “Motion planning and control for mobile robot navigation using machine learning: a survey,” 2022. [Online]. Available: https://arxiv.org/abs/2011.13112 [2] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra, “Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames,” 2020. [Online]. Available: https://arxiv.org/abs/1911.00357 [3] G. Vecchio, R. E. Sarpietro, F. Cancelliere, S. Palazzo, D. C. Guastella, A. Strano, I. Carlucho, G. Muscato, S. V. Albrecht, and C. Spampinato, “Midgard: A robot navigation simulator for outdoor unstructured environments,” in European Robotics Forum 2024. Cham: Springer Nature Switzerland, 2024, pp. 120–125. [4] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” Journal of Machine Learning Research, vol. 22, no. 268, pp. 1–8, 2021. [Online]. Available: http://jmlr.org/papers/v22/201364.html 216