Polynomial Regression-Augmented End-to-End Planner for Trajectory Prediction
Abstract
This project explores end-to-end trajectory planning for autonomous vehicles, developing a model that maps raw sensor data to future motion commands on the nuPlan dataset. The work details the model's progression across three milestones, beginning with a baseline architecture that suffered from weak generalization. The model was subsequently enhanced in simulation with multitask learning for depth and segmentation and, most critically, by incorporating a trajectory generated by polynomial regression as an input feature to significantly boost performance. For the final stage, the model was adapted for sim-to-real generalization by removing the auxiliary tasks and applying aggressive data augmentation. This refined, single-task architecture fuses visual features from an EfficientNet-B0 backbone with motion history using cross-modal attention, achieving a final public Average Displacement Error (ADE) of 1.14062 m.
Full text
Polynomial Regression–Augmented End-toEnd Planner for Trajectory Prediction Saúl Fenollosa, Filippo Quadri CIVIL-459 Deep learning for autonomous vehicles Data sample Trajectory prediction Depth prediction Segmentation prediction Planner model. :class TrajectoryPredictor(.) ‘camera’ ‘history’ ‘history ‘speed’ ‘acceleration’ ‘past_driving_commands’ ‘tendency’ ‘trajectories’ Visual encoder = : blackbone efficientnet_b0(.) History transformer= : history_transformer Tra n sform e rE nc od e r (.) H istory encoder . motion_encoder = . :nn Sequential(.) command_encoder = . :nn Sequential(.) combined_encoder = . :nn Sequential(.) M ul ti-h ead a tt en ti on = . : cross_a tt n nn Multih e adAtt en tio n(.) P redic ti on h ead = . : p re d _hea d nn S e qu en tial (.) S e g menta ti on decoder = . : se g _ d eco d er nn S e qu en tial (.) D e p t h decoder= . : d e p th_ d eco d er nn S e qu en tial (.) I / II / III I / II / III I / II / III I / II / III II / III II II Train fi les Train dataset Data generator. :class DrivingDataset (.) Validation fi les Validation dataset def : regression_fro _history(.) def : get_past_drivin _commands(.) def :get_speed(.) def :get_acceleration(.) ‘depth’ ‘speed’ ‘acceleration’ ‘trajectories’ ‘past _ driving _ commands’ ‘general _ tendency’ ‘semantic _ la b el’ ‘history’ ‘camera’ II / III II / III II / III I / II / III I N T RO D U CTI ON In this project, we explore end-to-end trajectory planning for autonomous vehicles, directly mapping raw sensor data to future motion commands using deep neural networks. We progressively enhance our model across three milestones: Working with a curated subset of the nuPlan dataset, which consists of simulated driving scenarios with camera images, past vehicle trajectories, and annotated future paths — key distributional insights are shown below. MILESTONE Basic planning MILESTONE S im - to -R eal generali z ation MILESTONE P erception - aware planning R E L A T E D W OR K S H . -S . C hoi, J . J eong, Y . H . C ho, K . -J . Y oon, and J . -H . K im, “C ognitive T rans F user: S emantics - guided T ransformer - based S ensor F usion for Improved Waypoint P rediction, ” in P roc. R obot Intelligence T echnology and A pplications (R i TA) , 2023 ( ar X iv: 2308 . 02126) I. F reeman, K . Z hao, and A . K ummert, “P olynomial T rajectory P redictions for Improved L earning P erformance, ” in P roc. I EEE Int. C onf. on Image P rocessing ( I C I P) , 2021 , pp. 3313–3317 Y . F an, Y . L i, and S . Wang, “R isk -A ware S elf -C onsistent Imitation L earning for T rajectory P lanning in A utonomous D riving, ” in P roc. E uropean C onf. on C omputer V ision (ECCV) , 2024 ( to appear ) . ME T H O D Visual encode Pretrained EfficientNet B0 H istory encode MLPs & Transformers Cross-modal fusio Multi-head attention between image and motion encodings Trajectory hea Prediction of [x, y, heading ] o v er 6 0 time ste p s A u x iliary tas k S emantic segmentation and de p th decoders Driving commands Depth prediction ... Trajectory prediction R esidual connections H istory features R GB camera image T he diagram below summari z es the core components and fl ow of our proposed method. Work fl ow M ilestone I: D evelop a baseline model for trajectory prediction M ilestone II: E nhance and adapt the model for be tt er generali z ation in simulation M ilestone III: R e fi ne the model to retain simulation performance on real - world data. M IL E ST ON E I MOD E Visual Backbone: EfficientNet-B History features: LSTM + MLP + MultiheadAttentio Additional inputs: general_tendency, past_driving_command ISSU E Overfitting and weak generalization. M IL E ST ON E II U P G RA D E Multitask learning: Depth & Segmentatio Loss function tuning and inclusion of auxiliary losse Residual Connection + Relative Position Support + Further inputs (speed, acceleration Curve prediction remains inaccurate New input added: Polynomial Regression Trajectory + noise (to allow adaptability LSTM → MLP P olynomial predictions were shown to cover most samples with low ADE s. M IL E ST ON E III SU B T L E CH A NG E Removed Multitask learning suppor Aggressive data augmentatio Color correctio Noise additio Sim-toR eal generali z ation M otion blu P erspective change R E S U LTS S teady improvement in the ADE scor D i ff erent approaches applied to boost the performances and reduce the over - fi tti n R eal data outperforms synthetic data, even if the model is trained on synthetic dat A s seen in M ilestone II section, regression achieved good ADE score even without DL EXAMPLES OF PREDICTIONS R E F E R E N C E S M O D E L A R C H IT E CT UR E M ilestone I M ilestone II M ilestone III P hillips, J ., M artine z , J ., B â rsan, I. A ., C asas, S ., S adat, A ., & U rtasun, R . (2021) . D eep M ulti -T ask L earning for J oint L ocali z ation, P erception, and P rediction. ar X iv [ C s. CV ] . R etrieved from h tt p: // arxiv.org / abs / 2101 . 0672 H u, Y ., Y ang, J ., C hen, L ., L i, K ., S ima, C ., Z hu, X ., … L i, H . (2023) . P lanning - oriented A utonomous D riving. ar X iv [ C s. CV ] . R etrieved from h tt p: // arxiv.org / abs / 2212 . 101 5 Z hou, Z ., Y e, L ., Wang, J ., Wu, K ., & L u, K . (2022) . H i VT : H ierarchical V ector T ransformer for M ulti - A gent M otion P rediction. 2022 I EEE / CVF C onference on C omputer V ision and P a tt ern R ecognition (CVPR) , 8813–8823 . doi: 10 . 110 9/ CVPR 5 2688 . 2022 . 00862