[NeurIPS 2023] AD-PT: Autonomous Driving Pre-Training
Large-scale Point Cloud Dataset

Abstract

It is a long-term vision for Autonomous Driving (AD) community that the perception models can learn from a large-scale point cloud dataset, to obtain unified representations that can achieve promising results on different tasks or benchmarks. Previous works mainly focus on the self-supervised pre-training pipeline, meaning that they perform the pre-training and fine-tuning on the same benchmark, which is difficult to attain the performance scalability and cross-dataset application for the pre-training checkpoint. In this paper, for the first time, we are committed to building a large-scale pre-training point-cloud dataset with diverse data distribution, and meanwhile learning generalizable representations from such a diverse pre-training dataset. We formulate the point-cloud pre-training task as a semi-supervised problem, which leverages the few-shot labeled and massive unlabeled point-cloud data to generate the unified backbone representations that can be directly applied to many baseline models and benchmarks, decoupling the AD-related pre-training process and downstream fine-tuning task. During the period of backbone pre-training, by enhancing the scene- and instance-level distribution diversity and exploiting the backbone's ability to learn from unknown instances, we achieve significant performance gains on a series of downstream perception benchmarks including Waymo, nuScenes, and KITTI, under different baseline models like PV-RCNN++, SECOND, CenterPoint.

Drawback of previous 3D pre-training methods

Differences between previous pre-training paradigm and the proposed AD-PT paradigm.

Framework

The overview of the proposed AD-PT. By leveraging the proposed method to train on the unified large-scale point cloud dataset, we can obtain well-generalized pre-training parameters that can be applied to multiple datasets and support different baseline detectors.

Data preparation and statistical distribution using re-scaling

Overall dataset preparation procedure.

Visualization of re-sampling

Visualization of point-to-beam playback re-sampling.

Results on Waymo dataset

Results on nuScenes dataset

Bibtex

                    
@inproceedings{yuan2023ad-pt,
    title={AD-PT: Autonomous Driving Pre-Training with Large-scale Point Cloud Dataset},
    author={Yuan, Jiakang and Zhang, Bo and Yan, Xiangchao and Chen, Tao and Shi, Botian and Li, Yikang and Qiao, Yu},
    booktitle={Advances in Neural Information Processing Systems},
    year={2023}
}