A Comparative Study of DQN Training Methods on the Variable-Length CartPole Problem

A Comparative Study of DQN Training Methods on the Variable-Length CartPole Problem Kayra Özdemir Vrije Universiteit Amsterdam Rohan Menon Vrije Universiteit Amsterdam , , . ABSTRACT This study investigates how different training strategies for a reinforcement learning, Deep Q-Network (DQN) agent performs on a modified CartPole environment. This environment is modified to vary the pole length systematically, a key determinant of state dynamics. The objective is to assess how learning strategies influence agent stability, generalization, and robustness under these changing physical conditions. Three models were evaluated and compared: a baseline DQN with randomized pole lengths, a Double DQN (DDQN) variant, and a Curriculum Learning (CL) strategy. Each model was trained with the same neural net architectures (repeated twice for the DDQN) and hyperparameters, differing only in training logic. Evaluation across 30 pole lengths and multiple randomized test runs measured average rewards, success rates, and generalization correlations. Results demonstrate that the curriculum learning approach achieves the highest total and average rewards, perfect success rates, and improved robustness to shorter poles, while Double DQN and the baseline show lower adaptability. These findings support the efficacy of curriculum-based training for reinforcement learning under variable environment dynamics. 1 INTRODUCTION Generalization across changing physical environments remains a consistent challenge in reinforcement learning (RL). Training dynamic policies that can safely and accurately execute actions in situations where the underlying environment itself is changing, is one that is extremely important. As the physical world is constantly changing, and agents need to be able to adapt to handle state spaces that it may not have seen or had much experience with. The motivation stems from prior studies showing that RL agents trained under static conditions often fail to adapt to unseen configurations. Conventional DQN networks have proven to perform well on the conventional cartpole task, where a simple feedforward network performs very well as a value function estimator in complex environments. However, it can have it’s limits in more dynamic environments where they may not always generalize to unseen or unfamiliar states. Curriculum Learning (CL) has been shown to enhance learning efficiency and cross-task transfer through progressive task difficulty [ 4 , 6 , 10 ], while Double DQN mitigates Q-value overestimation and stabilizes convergence [ 8 ]. This work aims to answer the central research question: How do different training schedules and network update mechanisms influence the generalization capability of Deep Q-Networks in environments with changing physical parameters? 2 METHODOLOGY 2.1 Experimental Setup All experiments were conducted on the CartPole-v1 environment provided through the OpenAI Gym API [ 3 ], with the pole length acting as the varying physical parameter. Each state in the gym cartpole environment consists of an array which contains the cart position, cart velocity, pole angle and pole angular velocity. All of these were kept constant as required. Pole lengths ranged from 0.4 m to 1.8 m, evaluated across 30 evenly spaced values. During testing, each pole length was simulated 10 times under identical stochastic conditions with the exception of curriculum learning where the agent received poles in increasing length. Each DQN agent operated under the same simple feedforward neural network. This consisted of a multilayer perceptron of 4 neurons (to represent the state space), followed by two hidden layers of 128 neurons and a ReLU activation function followed by an output dimension of 2 neurons. Where 0 moves the cart left, and 1 moves the cart right. Hyperparameters were also made to be consistent across models. The chosen hyperparameters were: •Learning Rate: 1∗10−3 •Gamma: 0.99 •𝜖Start = 1.0, 𝜖End = 0.01, 𝜖Decay = 50000 •Max Steps: 500 •Seed: 0 2.2 Difficulty of Pole Lengths Longer poles are physically easier to balance due to their increased moment of inertia. A longer pole rotates more slowly in response to small angular deviations, meaning the cart has more time to correct the pole’s fall before the angle grows too large. In contrast, shorter poles have a smaller moment of inertia and therefore fall much more quickly, requiring faster and more precise corrective actions. This aligns with classical mechanics: the angular acceleration of an inverted pendulum is inversely proportional to its length, making long poles dynamically more forgiving and short poles highly unstable. Consequently, pole length naturally induces a meaningful difficulty gradient in the CartPole environment. 2.3 Baseline DQN The baseline model implements the standard Deep Q-Network (DQN) algorithm, which approximates the optimal action-value function Q ∗(𝑠, 𝑎) using a neural network parameterized by 𝜃 . At each time step t, the network predicts the expected discounted return for all possible actions, and the policy selects actions via a 𝜖 -greedy rule to balance exploration and exploitation. The target for the Q-update follows the Bellman equation[1]: , , Kayra Özdemir and Rohan Menon 𝑦𝑡=𝑟𝑡+𝛾max 𝑎′𝑄(𝑠𝑡+1, 𝑎′;𝜃−) where 𝜃− is the parameters of the periodically updated target network. The temporal-difference (TD) error is minimized by gradient descent on the mean squared loss (yt−Q(st,at;𝜃))2. The baseline model trained a standard DQN with randomized pole lengths sampled uniformly at each episode. This means that it did not receive the pole lengths in any specific order, rather the environment gave it pole lengths between 0.4 and 1.8 randomly (controlled by a seed). This simulates a naive generalist approach without structured difficulty progression. All other components (experience replay, target updates, loss computation; their parameters) followed the standard DQN design. 2.4 Double DQN Algorithm DQN’s can be known to be prone to overestimation bias, which comes from the max operation of the Bellman target. In standard DQN implementations, the same network both selects and evaluates the next action. This can often lead to inflated Q-values and suboptimal learning convergence. The Double DQN extends this system to address this overestimation bias[13]. Double DQN seperates these roles by using the online network for action selection and the target network for action evaluation. The target value is modified to: 𝑦Double 𝑡=𝑟𝑡+𝛾𝑄 (𝑠𝑡+1,arg max 𝑎′𝑄(𝑠𝑡+1, 𝑎′;𝜃);𝜃−) This change is proposed to prevent the network from overestimating action values, improving smooth convergence and training stability. In this experiment, the DDQN shares the same architecture, replay buffer, and exploration schedule as the baseline. The only modification lies in the update rule where we have separated the action selection and evaluation. By keeping all other parameters constant, the effect of the decoupled target computation can be isolated and directly compared to both the baseline and curriculumbased training. 2.5 Curriculum Learning Strategy The Curriculum Learning (CL) [ 11 ] model follows a defined paced schedule in which the pole length decreases progressively as the agent’s performance improves according to the curriculum. The pole lengths were defined as: 𝐿=[1.8,1.6,1.4,1.2,1.0,0.8,0.6,0.5,0.45,0.4] The agent starts with the longest (easiest) pole in the curriculum L0 and transitions to shorter poles L1,L2, ... when its moving average return R100 over the past 100 episodes exceeds a fixed threshold 𝜏=125 , and the amount of episodes spent for the current length eLexceeds a minimum episode count (min_𝜖=10): 𝑙𝑒𝑛𝑔𝑡ℎ =(𝐿𝑖+1,if 𝑅100 >𝜏and 𝑒𝐿>min_𝜖 𝐿𝑖,otherwise. This logic ensures that exposure to higher task complexity occurs only after stable control behavior has emerged for the current length, aligning the agent’s learning pace with its performance capacity. In further detail, this makes the model a self-paced curriculum learning strategy [ 9 ], where the length transitions are determined by performance rather than fixed in time [ 4 , 6 ]. All the other components of the DQN framework (experience replay, target updates, exploration schedule, loss computation, etc.) remain unchanged. This isolates the effect of the curriculum mechanism, and the CL strategy is therefore only implemented in the train module. The parameters of CL are: pole lengths array L (defined/set to be the values given above) performance threshold (set to 125.0) minimum episode threshold min_eps (set to 10) all the other parameters from the baseline model remain unchanged. The performance threshold 𝜏=125 defines the point at which the agent is considered to have achieved stable control before progressing to harder tasks. This value provides enough margin above early, unstable performance while keeping training momentum. The minimum episode limit ( (min_𝜖=10) ) prevents transitions that are too frequent, and ensures at least 1% of the training is spent on each difficulty before advancing. Together, these parameters balance stability and progression within the curriculum. 2.6 Data Collection In order to conduct rigorous analysis, a more detailed logging system was added to the testing file test_script, without modifying the original logic of the testing environment or the original data format produced directly from the assignment skeleton. For each pole length, the current state data is stored in a DataFrame. After each pole length is tested 10 times, the DataFrame is converted to a .csv file test_results, with the columns being: the model name, the (current) length of the pole, the current episode, the score, and a success flag (whether the score reached 500 or not). 2.7 Evaluation & Statistical Testing Model evaluation was performed to quantify both overall performance and statistical significance across shared task conditions. All agents were tested under identical pole length configurations, enabling a repeated-measures experimental design. Evaluation was conducted at two levels, to capture both cross-environment and per-environment data: • Global (per-model) metrics summarize overall generalization and robustness [ 10 ]. The global metrics were computed using the test_results data. These include the total reward across all pole lengths, the average mean reward across all pole lengths, stability (standard deviation of means), success rate (percentage of lengths with mean 500), and the Pearson correlation between pole length and mean reward, a metric indicating whether performance improves for longer poles or deteriorates for shorter ones [ 6 ]. The best and worst lengths are also reported to identify the easiest and hardest configurations directly, although further detail is included in the local metrics. • Local (per-length) metrics capture the per-length and standard deviations of the achieved rewards, for further analysis on the specific behavior each model has per-length. The local metrics were computed using the data from the experiment_results file, given by the assignment skeleton. A Comparative Study of DQN Training Methods on the Variable-Length CartPole Problem , , To assess whether model differences are statistically significant, non-parametric, rank-based tests were applied to the episode-level data. These tests were chosen specifically because reinforcement learning performance distributions often violate normality and equal-variance assumptions, rendering classical parametric ANOVA invalid [ 4 ]. Following established guidelines for algorithm comparison in machine learning research [ 4 ], and reinforced by recent reinforcement learning evaluation studies [ 8 ], the following procedure was adopted: (1) Friedman test is the omnibus test (an omnibus test identifies if at least one group differs, without specifying the group [ 2 ]) for detecting significant differences among three or more related models [5]. (2) Wilcoxon signed-rank tests are pairwise post-hoc comparisons (tests done after the omnibus result to pinpoint which pairs differ significantly [ 12 ]) between specific model pairs [14]. (3) Holm’s sequentially rejective correction controls the family-wise error rate across multiple comparisons (adjusts the p-values of the results sequentially, reducing false positives while real effects remain detectable), which is the chance of finding a significant result by coincidence or noise [7]. This protocol follows best practices for comparing reinforcement learning algorithms when the experimental conditions are the same [ 4 ]. The Friedman test detects overall model differences, Wilcoxon tests identify pairwise contrasts, and Holm’s correction controls false positives, ensuring statistically reliable and interpretable comparisons [5, 7, 8, 14]. 3 RESULTS & DISCUSSION Table 1: Model Comparative Results (Global) Model Average Reward Standard Deviation Success Rate (%) Pearson Correlation Baseline 744.7 573.2 66 0.17 CL 21657.2 10661.9 100 0.47 DDQN 4390.3 5017.9 53 -0.66 Table 2: Model Statistical Results Test Comparison Statistic 𝑝Holm Significant Friedman All models 43.93 2.88e-10 Yes Wilcoxon Baseline vs. CL 0.01.58e-05 Yes Baseline vs. DDQN 171.00.2 No CL vs. DDQN 0.04.69e-06 Yes Model evaluation focused on identifying the agents’ ability to generalize across the 30 given pole lengths, using global metrics of performance and robustness, as summarized in Table 1. The Curriculum Learning (CL) agent significantly outperformed the other two models across all relevant metrics. The CL agent achieved an Average Reward of 21657.2, which is more than 29 times the reward of the Baseline DQN (744.7) and nearly 5 times the reward of the Double DQN (4390.3). This superior generalization is definitively demonstrated by the CL model’s 100% Success Rate, indicating it was able to successfully stabilize the pole for every single one of the 30 tested pole lengths. In contrast, the Baseline DQN showed moderate generalization, achieving a 66% Success Rate. The Double DQN (DDQN) performed the worst in terms of consistency, with a Success Rate of only 53%. Although the DDQN’s Average Reward (4390.3) was substantially higher than the Baseline, this performance was highly unstable, as suggested by its large Standard Deviation (5017.9) relative to its mean. The Friedman test revealed a statistically significant difference among the three DQN variants (H=43.93, pholm =2.89), confirming that at least one model performed differently under shared evaluation conditions. Post-hoc Wilcoxon signed-rank tests with Holm correction further clarified these differences. The Curriculum DQN significantly outperformed both the Baseline DQN (W=0, pholm =1.58) and the Double DQN (W=171, pholm =0.2), while no significant difference was found between the Baseline and Double DQN (W=0, pholm =4.7). These results indicate that the curriculum learning strategy provided a statistically confirmed improvement in performance consistency and generalization over pole lengths, compared to the other models [5, 7, 8, 14]. The Curriculum Learning agent’s strong performance can be attributed to its progressive training schedule (Appendix A, Figure 1). By starting with the longest pole and only advancing to shorter configurations after achieving stable control, the CL agent built a foundational policy that successfully generalized across the full range of parameters, confirming that structuring difficulty progression is essential for transferring learning. In contrast, the Baseline DQN, utilizing a "naive generalist approach" with randomized pole lengths, failed to build this foundation, resulting in the lowest overall performance. The Double DQN (DDQN), while theoretically designed to stabilize learning by limiting Q-value overestimation, showed no statistically significant improvement over the Baseline. It had a higher average reward, but a lower episode completion rate. This indicates that just improving the network update rule, without having a structured training schedule, was not enough to improve generalization across dynamically changing physical environments. It also was the only example with a negative Pearson Correlation, highlighting that it was finding the easier pole lengths harder, reinforcing the hypothesis that separated target computation did not relate to robust generalization. 4 CONCLUSION & FUTURE WORK This work investigated the influence of training methods and network update processes on the generalization capabilities of Deep Q-Networks (DQN) in a dynamic CartPole environment, where the pole length was dynamically varied. The results conclusively answer the question and show that the method of training organization is the dominant factor in maintaining generalization robustness. The Curriculum Learning strategy significantly statistically outperformed both the DDQN and the baseline DQN networks. Conversely, while the DDQN architecture is theoretically stronger in stabilizing convergence to the baseline DQN, it struggled to show , , Kayra Özdemir and Rohan Menon a statistically significant improvement. This suggests that for scenarios requiring adaptation to changing physical environments, a structured difficulty progression that builds skills incrementally is more effective than simple network update modifications. The findings of this study open possible areas for future research. One possible avenue could be evaluating a combined model (CLDDQN) to determine if stabilizing Q-value estimation can further enhance the generalization achieved by the curriculum strategy. There is also room to test the curriculum learning strategy in more challenging, high-dimensional continuous control tasks, such as robotic simulations, to confirm real-world scalability. A PLOTS Figure 1: Model Comparative Results (Local) Figure 2: Baseline Results B GITHUB REPOSITORY The code for all the models and data analysis can be found in the following GitHub repository: https://github.com/storks-amsterdam/vu-reinforcement-learning The trained models are available under the following directory, as .pt files: vu-reinforcement-learning/src/networks Figure 3: Curriculum Results Figure 4: Double DQN Results Figure 5: CartPole Simulator A Comparative Study of DQN Training Methods on the Variable-Length CartPole Problem , , Figure 6: Full List of Network Hyperparameters , , Kayra Özdemir and Rohan Menon REFERENCES [1] Richard Bellman. Dynamic programming. Princeton University Press, 1957. [2] Opeoluwa Bfadeyi. ROBUSTNESS AND COMPARATIVE STATISTICAL POWER OF THE REPEATED MEASURES ANOVA AND FRIEDMAN TEST WITH REAL DATA. PhD thesis, Wayne State University, 2021. [3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016. [4] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(1):1–30, 2006. [5] Milton Friedman. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200):675–701, 1937. [6] Hiroki Furuta, Tatsuya Matsushima, Tadashi Kozuno, Yutaka Matsuo, Sergey Levine, Ofir Nachum, and Shixiang Shane Gu. Policy information capacity: Information-theoretic measure for task complexity in deep reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 3541–3552. PMLR, July 18–24 2021. [7] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979. [8] H. Huang. Rl in video games. Technical report, Vanderbilt University, —. Technical report / lecture notes. [9] Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G. Hauptmann. Self-paced curriculum learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 2694–2700, 2015. [10] Ezgi Korkmaz. A survey analyzing generalization in deep reinforcement learning. arXiv preprint arXiv:2401.02349, 2024. [11] Nilaksh, Abhishek Ranjan, Shreenabh Agrawal, Aayush Jain, Pushpak Jagtap, and Shishir Kolathaya. Barrier functions inspired reward shaping for reinforcement learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), page 10807–10813. IEEE, May 2024. [12] G. D. Ruxton and G. Beauchamp. Some method / title —. International Society for Behavioral Ecology, 2008. [13] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. CoRR, abs/1509.06461, 2015. [14] Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80–83, 1945.