Pedestrian Trajectory Prediction : Multimodal Sensor Fusion for Road Crossing

Abstract

Pedestrian safety is one of the most integral research areas in the topic of autonomous vehicles (AVs). Extensive research has been conducted on the topic of predicting pedestrian trajectories, yet most of this research focuses on monosensor modalities. Meanwhile, the researches that focus on multimodal and sensor fusion approaches mainly focus on detecting pedestrians and not predicting their intended trajectory. In this research, we propose a multimodal approach for predicting pedestrian trajectories, which addresses the shortcomings of unimodal trajectory prediction by fusing LiDAR, radar, and camera data to provide a more accurate prediction of pedestrian trajectories. To achieve this, we create a pipeline that receives the inputs of Ouster OS1 LiDAR, Navtech RAS3 Radar, and Intel RealSense D415, temporally synchronizes them, calibrates them in a common coordinate frame, and fuses them into a dataset. Pedestrian annotations are obtained using PointRCNN combined with a nearest-neighbor tracker to assign consistent IDs across frames. We then evaluate state-of-the-art trajectory prediction models on the resulting dataset. Results show that a zero-shot Social-STGCNN baseline yields 0.51 ADE / 0.76 FDE, while a trained Wayformer model achieves 2.95 ADE / 2.31 FDE.