VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY FACULTY OF COMPUTER SCIENCE & ENGINEERING ——————– * ——————— GRADUATE THESIS 3D OBJECT POSE DETECTION FROM IMAGE Major: Computer Science Council: COMPUTER SCIENCE 02 Supervisor: Dr. NGUYEN DUC DUNG Reviewer: Msc. LUU QUANG HUAN —o0o— Student: BUI VIET MINH QUAN (1710259) HO CHI MINH CITY, 08/2021 Declaration We hereby declare that this thesis titled ‘3D OBJECT POSE DETECTION FROM IMAGE’ and the work presented in it are our own. We confirm that: • This work was done wholly or mainly while in candidature for a degree at this University.
• Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated. • Where we have consulted the published work of others, this is always clearly attributed. • Where we have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely our own work.
• We have acknowledged all main sources of help. • Where the thesis is based on work done by ourselves jointly with others, we have made clear exactly what was done by others and what we have contributed ourselves. Acknowledgments First of all, we would like to express our greatest respect and gratitude to our supervisors, Dr. Nguyen Duc Dung and Dr.
Pham Hoang Anh for their profound trust, support and guidance. They have spent so much time helping us revise this work rigorously. Furthermore, they gave us many good advices and strengthened our motivation when we were struggling. The enthusiasm and energy in them has inspired us so much in our work.
We surely could not complete this thesis without them. Thank our beloved colleagues, friends and all the people who have encouraged us during this stage of our life. Thank HCMC University of Technology and the Faculty of Computer Science and Engineering for giving us this wonderful experience. Abstract Monocular 3D object detection has recently become prevalent in autonomous driving and nav- igation applications due to its cost-efficiency and easy-to-embed to existent vehicles.
The most challenging task in monocular vision is to estimate a reliable object’s location cause of the lack of depth information in RGB images. Many methods tackle this ill-posed problem by directly regressing the object’s depth or take the depth map as a supplement input to enhance the model’s results. However, the performance relies heavily on the estimated depth map quality, which is bias to the training data. In this work, we propose depth-adaptive convolution to replace the tra- ditional 2D convolution to deal with the divergent context of the image’s features.
This lead to significant improvement in both training convergence and testing accuracy. Second, we propose a ground plane model that utilizes geometric constraints in the pose estimation process. With the new method, named GAC3D, we achieve better detection results. We demonstrate our ap- proach on the KITTI 3D Object Detection benchmark, which outperforms existing monocular methods.
Benefiting from this simple structure, ours is much faster than many state-of-the-art methods and enables real-time inference. Table of contents 1 Introduction 1 1.1 The Perspective Projection Matrix .2 The Intrinsic Parameters and the Normalized Camera .4 Feed-forward Neural Network .5 Convolutional Neural Network .6 Least Squares Problems and Singular Value Decomposition .1 Least Squares Problems .2 Singular Value Decomposition and Pseudo-inverse Matrix .7 CenterNet: Object Detector by Keypoints .1 Neural Network Layout. 15 3 Datasets and Metrics 16 3.2 Definitions of various metrics .1 LiDAR-based 3D object detection .2 Monocular 3D object detection using representation transformation .3 Monocular 3D object detection using anchor-based detector .4 Monocular 3D object detection using center-based detector .1 Base Detection Network .5 Pseudo-contact Point Head .2 Detection Network With Depth Adaptive Convolution .1 Depth Adaptive Convolution .2 Depth Adaptive Detection Head .3 Variant Guidance For Depth Adaptive Convolution .3 End-to-End Depth Guidance .5 Geometric Ground-Guide Module .1 Object’s Pseudo-position .1 Image Preprocessing and Data Augmentation .1 Accumulated impact of our proposed methods .2 Evaluation on Depth Adaptive Convolution .3 Evaluation on the impact of depth estimation quality on Adaptive Con- volution .4 Abnormal detection cases of KITTI dataset. 62 8 Conclusion 71 9 More qualitative results 73 10 Some failure cases 75 11 Implementation of the detection networks 76 List of tables 2.1 Output layer activation functions and loss functions .1 Software specification of the training machine .1 KITTI Object Detection benchmark of our GAC3D method.2 KITTI Object Detection benchmark of our GAC3D-E2E-Lite method.3 KITTI Object Detection benchmark of our GAC3D-E2E method.4 Comparative results on the KITTI 3D object detection test set of the Car category.5 Comparative results of the Pedestrian and Cyclist on KITTI test set.6 Evaluation on accumulated improvement of our proposed methods on KITTI val set .7 Impact of Geometric Ground-Guide Module for 3D object detection on the KITTI val set.8 Comparisons of different depth estimation quality for 3D object detection on KITTI val set .1 The implementation of the detection network with ResNet-18 backbone.2 The implementation of the detection network with DLA-34 backbone.
79 iii List of figures 1.1 An example of occluded object in driving scenario .1 Front-view-camera image and LiDAR 3D point cloud from a sample of KITTI dataset[20].2 Image formation in a pinhole camera.3 The pinhole camera model.4 The optical axis, focal plane, and retinal plane.5 The coordinate system.6 Changing coordinate systems in the retinal plane.7 A neural network with 1 hidden layer.1 Examples from the KITTI dataset (from left color camera)[19].2 Recording platform of KITTI dataset[19].1 Network architecture of PointNet[50].2 PV-RCNN[57] proposed architecture.3 CaDDN[52] proposed architecture.4 Predefined 3D anchors in M3D-RPN[6].5 M3D-RPN[6] proposed architecture and the depth aware convolution.6 D4LCN[15] proposed architecture.7 Ground aware convolution from [39].8 Network structure of SMOKE[41].9 Proposed architecture of RTM3D[34].10 KM3D-Net[33] proposed architecture.1 Overview of the proposed network .2 Overview of the baseline network .3 The layers configuration of a residual block.4 The ground truth heatmap of 2D bounding box’s center.5 Illustration of keypoints regression.6 The projected location on 2D image plane of the pseudo-contact point.7 Equal egocentric angles (left) and equal allocentric angles (right).8 The moving car has the same egocentric (global) orientation (moving forward) in 3 frames while the allocentric (local) orientation with respect to the camera is changed.9 Egocentric (green color) and allocentric (orange color) angles in the bird’s- eye-view. Red arrow indicates the heading of the car while blue arrow is the ray between the origin and the car’s center .10 The orientation decomposition. We decompose the observation angle into three components: axis classification, heading classification, and relative offset re- gression.11 The default dimension setting of KITTI dataset and the impact of visual appear- ance of object on dimension regression.12 Illustration of 3D confidence outputs. Green: prediction, red: ground truth.
The further car with less accurate 3D pose estimation yields lower 3D confidence score.13 Depth Adaptive Convolution Detection Head .14 Illustration of depth estimations guiding the depth adaptive convolution .15 Depth Adaptive Convolution Detection Head With Variant Guidance .16 The architecture of our detection network with end-to-end depth estimation .17 Illustration of ground truth (red color) and predicted (blue color) 3D bounding box in bird’s-eye-view.18 The depth of the ground plane generated from the extrinsic information of cam- era.19 Object’s pseudo-position P and related terms in the inference process using camera model and ground plane model.1 Color jittering augmentation .2 Horizontal flipping augmentation .3 Scaling and shifting augmentation .1 Illustration of unlabeled cases in KITTI val set .2 Visualization of the objects’ 2D centers heatmap from the center head of the detection network. Left: The monocular image from KITTI val split. Right: The corresponding detected heatmap of objects’ 2D centers for class Car.3 Visualization of the depth branch’s output. Left: The monocular image from KITTI val split.
Right: The corresponding end-to-end depth estimation from the depth branch.4 Visualization of the 2D projected objects’ keypoints prediction. Top: The cropped region of cars from monocular image. Bottom: The nine projected keypoints es- timated from the keypoints head.5 Detailed visualization of our 3D detection result.6 Qualitative illustration of our monocular 3D detection results (left: val set, right: test set). Green color: our predictions, red color: ground truth, dot: projected 3D center, diagonal cross: heading of object .7 Illustration of multi-class detection on KITTI val set.
Green: Car, Yellow: Cy- clist, Purple: Pedestrian .8 Visualization of the impact of pseudo-position for refining object’s position. Red: groundtruth z-position, Green: predicted z-position.9 Visualization of the impact of pseudo-position for refining object’s position. Red: groundtruth z-position, Green: predicted z-position.10 Trajectories of the optimization process for each detection head with standard convolution and depth adaptive convolution operation.11 Abnormal detection cases from KITTI val set. Left: ground truth labels, right: our predictions.1 Visualization of the detection result for image ’000422’ of KITTI val set.2 Visualization of the detection result for image ’000527’ of KITTI val set.1 Failure case: objects in congested situation.2 Failure case: very distant object.3 Failure case: highly occluded and truncated objects.1 Introduction In recent years, with the evolution of the deep neural network in computer vision [25, 24, 8], we have seen various methods being proposed to resolve 2D object detection task [54, 23, 53, 67, 61], and achieve remarkable performance, which almost approach human visual perception.
Even so, in particular fields such as autonomous driving or infrastructure-less robot navigation, the demand for scene understanding, including the detailed 3D poses, identities, and scene con- text, is still high. Researchers pay attention to 3D object detection, especially in autonomous navigation applications. To obtain an accurate depth map of the environment, people adopt LiDAR sensors widely due to their reliable 3D point cloud acquired using laser technology. Such LiDAR-based systems [58, 29, 57, 22] achieve promising results, they also come with visible limitations, including high-cost sensors, hard to mount on vehicles, sparse and unstruc- tured data.
Therefore, an alternative solution using a singular RGB camera is required. It is far more affordable, versatile, and almost available on every modern vehicle. The main difficulty of image-based 3D object detection is the lack of depth information, which results in a signifi- cant gap performance compared to LiDAR-based methods. While stereo systems are available, we have to calibrate the cameras with relatively high accuracy.
Commercial cameras, such as Bumblebee, are compact systems with low cost. The quality of the obtained depth map, how- ever, is nowhere near the required standard for autonomous driving systems, especially in the outdoor environment. Estimating the depth via a single camera is a good choice since human also perceives the depth from 2D images. However, the accuracy of monocular depth estimation is not as good as ToF cameras like LiDAR.
That makes the 3D object detection task on a single camera more challenging and has brought much attention from the community. In this work, we propose a 3D object detection system based on a single camera view. We also demonstrate that our proposed framework can bridge the gap between LiDAR and image-based detectors. There are two main approaches in monocular 3D object detection: the representation trans- formation and the 2D convolutional neural network (CNN).
In the representation approach, the general idea is to imitate the 3D point clouds of LiDAR by estimating depth information from images. This depth map is then projected into 3D space to generate the pseudo-point clouds. With the pseudo point clouds, one can employ several algorithms that using LiDAR data to detect objects. However, the raw point clouds are sparse due to the range laser sensor’s phys- ical principle.
The pseudo point clouds, however, are considerably denser and depend heavily on the estimated depth map quality. Thus, applying object pose detection on these point clouds decreases the performance significantly. Besides, the pseudo-LiDAR methods consist of sev- eral separate steps, which usually require training the estimation model separately, leading to 1 INTRODUCTION Figure 1.