3D Panoptic Segmentation with Unsupervised Clustering for Visual Perception in Autonomous Driving

For the past decade, substantial progress has been achieved in the field of visual per ception for autonomous driving application thanks notably to the capabilities of deep learning techniques. This work aims to leverage stereovision and explore different methods, in particular unsupervised clustering approaches, to perform 3D panoptic segmentation for navigation purposes. The main contribution of this work consists in the development, test and validation of a novel framework in which geometric and semantic understanding of the scene are obtained separately at the pixel level. The combination of both for the extracted visual 2D information of the desired class provides a 3D sparse classified point cloud, which is used afterward for instance clustering. Preliminary tests of the baseline version of the framework for Vehicle objects were conducted on urban driving datasets. Results demonstrate for the first time the via bility for processing of this type of point cloud from visual data, and reveal improve ments areas. Specially, the importance of the boundary F-score in semantic seg mentation is highlighted for the first time in this application, with an increase up to 32 percentage point in this study. Additional contribution was made by applying distribution clustering as well as density based clustering for instance segmentation in a visual based 3D space representa tion. Results showed that DBSCAN was well suited for this application. As a result, it was proven that the presented framework can successfully provide genuine 3D profile map representation and localisation of vehicles in a urban environment from 2D visual information only. Furthermore, the mathematical formalisation of the link between DBSCAN’s param eter selection and camera projective geometry was presented as future work and a mean to demystify parameter selection.