Innovative deep learning solutions leveraging unlabeled and multi-source annotated datasets to segment and monitor change in large-scale 3D point cloud for Digital Twin Cities

A Digital Twin City (DTC) is a virtual representation of physical assets across their respective life cycles in a city. This is an emerging concept that has become the centre of attention for academia in more recent years. The rapid development of mobile LiDAR system (MLS) provides large-scale, accurate and affordable 3D point clouds (PCs) at a city-scale. These data offer high-quality geometric measurements but limited non-geometric semantics, such as object type and topological relationships. Therefore, semantic segmentation is a fundamental perception task for the massive datasets used in DTC creation.

In recent years, the development of deep neural networks (DNNs) has led to great success in 3D PC semantic segmentation while focusing on designing fully supervised networks for densely annotated data. However, such massive 3D point-wise annotation is time-consuming, labour-intensive, and error-prone. As a result, current effort is in developing efficient methods for creating more labels or learning from cheap supervision (i.e. little annotation or simulated data).

DTC involves 4D dynamic objects. Since LiDAR provides a snapshot of the DTC at a certain timestamp, there is a need to detect and understand changes in multi-temporal LiDAR datasets. Performing change detection in street-level PC streams is often a significantly complex task. One must expect a large variety of object shapes and appearances, numerous occlusion artifacts between the different objects, and a lack of accurate registration between the compared datasets. There is a crucial lack of studies using DNNs to address this problem. Furthermore, DTC creation should be addressed with a focus on automatic updating because a city is constantly changing. Existing urban 3D modelling methods are limited in this respect.

(Figure : Large-scale outdoor datasets (1-2 : uneven point density; 3 : significant occlusion; 4 : complex shape; 5 : noise))

Objectives

By extending our on-going research, the grand goal for this research is to design a deep learning-based framework for large-scale outdoor point clouds to accommodate the need for Digital Twin City creation and monitoring over time.

The intent is:

1) to develop innovative deep learning approaches to be less dependent on annotated data and better able to leverage datasets with different characteristics, even unlabelled or simulated ones;

2) to propose solutions to detect and monitor changes in urban environments at the point level;

3) to design new approaches able to learn the spatial and dynamic structures of urban environments in order to propose suitable and consistent 3D reconstructions.

Anticipated impact and significance of the project

This research program will be instrumental towards the advancement of knowledge on deep learning applied to large-scale outdoor PCs. Furthermore, it will contribute to 3D PCs contexts that are overlooked in the literature which is often dedicated to autonomous vehicles. More specifically, the proposed research program aims at tackling the challenges and problems related to learning in an unsupervised context, generalizable unsupervised approaches, change detection at the point level as well as learning 3D spatial representations from generative models. Even though there has recently been a strong interest and increased research work on URL applied to 3D PCs, this domain still lags far behind its counterparts in NLP and 2D computer vision tasks.

DTC is an emerging concept that has become the centre of attention for industry and academia. There is an increasing demand for 3D city models to support smart cities, autonomous driving, outdoor augmented reality, urban planning, construction 4.0 applications, to name a few. This research could lead to significant advancements in these fields by providing large-scale, accurate, semantically rich and affordable data at a city-scale as well as tools to track city assets over time.

Research team

  • Dr. Sylvie Daniel, Principal investigator, CRDIG, UL
  • Ph.D.: current open position for a full-time graduate student

Funding organisation

This project is financially supported by the Canadian Research Council NSERC.

Contact

N’hésitez pas à nous contacter pour toute question ou pour plus d’information à propos de ce projet.

Relancer l’entrepreneuriat local et l’activité économique en contexte... Chaire de recherche du Canada sur les villes et mobilités intelligentes et...