HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY LE VAN HUNG 3-D OBJECT DETECTIONS AND RECOGNITIONS: ASSISTING VISUALLY IMPAIRED PEOPLE Major: Computer Science Code: 9480101 DOCTORAL DISSERTATION OF COMPUTER SCIENCE SUPERVISORS: 1. Nguyen Thi Thuy Hanoi − 2018 HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY LE VAN HUNG 3-D OBJECT DETECTIONS AND RECOGNITIONS: ASSISTING VISUALLY IMPAIRED PEOPLE Major: Computer Science Code: 9480101 DOCTORAL DISSERTATION OF COMPUTER SCIENCE SUPERVISORS: 1. Nguyen Thi Thuy Hanoi − 2018 DECLARATION OF AUTHORSHIP I, Le Van Hung, declare that this dissertation titled, ”3-D Object Detections and Recognitions: Assisting Visually Impaired People in Daily Activities ”, and the works presented in it are my own. I confirm that: This work was done wholly or mainly while in candidature for a Ph.
research degree at Hanoi University of Science and Technology. Where any part of this thesis has previously been submitted for a degree or any other qualification at Hanoi University of Science and Technology or any other institution, this has been clearly stated. Where I have consulted the published work of others, this is always clearly at- tributed. Where I have quoted from the work of others, the source is always given.
With the exception of such quotations, this dissertation is entirely my own work. I have acknowledged all main sources of help. Where the dissertation is based on work done by myself jointly with others, I have made exactly what was done by others and what I have contributed myself. Hanoi, November 2018 PhD Student Le Van Hung SUPERVISORS Dr.
Vu Hai Assoc. Nguyen Thi Thuy i ACKNOWLEDGEMENT This dissertation was written during my doctoral course at International Research Institute Multimedia, Information, Communication and Applications (MICA), Hanoi University of Science and Technology (HUST). It is my great pleasure to thank all the people who supported me for completing this work. First, I would like to express my sincere gratitude to my advisors Dr.
Hai Vu and Assoc. Thi Thuy Nguyen for their continuous support, their patience, motivation, and immense knowledge. Their guidance helped me all the time of research and writing this dissertation. I could not imagine a better advisor and mentor for my Ph.
Besides my advisors, I would like to thank to Assoc. Thi-Lan Le, Assoc. Thanh-Hai Tran and members of Computer Vision Department at MICA Institute. The colleagues have assisted me a lot in my research process as well as they are co-authored in the published papers.
Moreover, the attention at scientific conferences has always been a great experience for me to receive many the useful comments. During my PhD course, I have received many supports from the Management Board of MICA Institute. My sincere thank to Prof. Yen Ngoc Pham, Prof.
Eric Castelli and Dr. Son Viet Nguyen, who gave me the opportunity to join research works, and gave me permission to joint to the laboratory in MICA Institute. Without their precious support, it has been being impossible to conduct this research. student of 911 program, I would like to thank this programme for financial support.
I also gratefully acknowledge the financial support for attending the conferences from Nafosted-FWO project (FWO.08) and VLIR project (ZEIN2012RIP19). I would like to thank the College of Statistics over the years both at my career work and outside of the work. Special thanks to my family, particularly, to my mother and father for all of their sacrifices that they have made on my behalf. I also would like to thank my beloved wife for everything she supported me.
Hanoi, November 2018 Ph. Student Le Van Hung ii CONTENTS DECLARATION OF AUTHORSHIP i ACKNOWLEDGEMENT ii CONTENTS v SYMBOLS vi LIST OF TABLES viii LIST OF FIGURES xvii 1 LITERATURE REVIEW 8 1.1 Aided-systems for supporting visually impaired people .1 Aided-systems for navigation services .2 Aided-systems for obstacle detection .3 Aided-systems for locating the interested objects in scenes .2 3-D object detection, recognition from a point cloud data .1 Appearance-based methods .2 Geometry-based methods .3 Datasets for 3-D object recognition .3 Fitting primitive shapes .1 Linear fitting algorithms .2 Robust estimation algorithms .3 RANdom SAmple Consensus (RANSAC) and its variations. 23 2 POINT CLOUD REPRESENTATION AND THE PROPOSED METHOD FOR TABLE PLANE DETECTION 24 2.1 Point cloud representations .1 Capturing data by a Microsoft Kinect sensor .2 Point cloud representation .2 The proposed method for table plane detection .3 The proposed method .1 The proposed framework .3 Table plane detection and extraction .1 Experimental setup and dataset collection .2 Table plane detection evaluation method .3 Separating the interested objects on the table plane .1 Coordinate system transformation .2 Separating table plane and the interested objects. 48 3 PRIMITIVE SHAPES ESTIMATION BY A NEW ROBUST ES- TIMATOR USING GEOMETRICAL CONSTRAINTS 51 3.1 Fitting primitive shapes by GCSAC .3 The proposed a new robust estimator .1 Overview of the proposed robust estimator (GCSAC) .2 Geometrical analyses and constraints for qualifying good samples .4 Experimental results of robust estimator .1 Evaluation datasets of robust estimator .2 Evaluation measurements of robust estimator .3 Evaluation results of a new robust estimator .2 Fitting objects using the context and geometrical constraints .1 The proposed method of finding objects using the context and geometrical constraints .1 Model verification using contextual constraints .2 Experimental results of finding objects using the context and geometrical constraints .1 Descriptions of the datasets for evaluation .3 Results of finding objects using the context and geo- metrical constraints.
85 iv 4 DETECTION AND ESTIMATION OF A 3-D OBJECT MODEL FOR A REAL APPLICATION 86 4.1 A Comparative study on 3-D object detection .3 Three different approaches for 3-D objects detection in a complex scene .1 Geometry-based method for Primitive Shape detection Method (PSM) .2 Combination of Clustering objects and Viewpoint Features Histogram, GCSAC for estimating 3-D full object mod- els (CVFGS) .3 Combination of Deep Learning based and GCSAC for estimating 3-D full object models (DLGS) .3 Setup parameters in the evaluations .2 Deploying an aided-system for visually impaired people .1 Environment and material setup for the evaluation .2 Pre-built script .3 Performances of the real system .1 Evaluation of finding 3-D objects .4 Evaluation of usability and discussion. 118 5 CONCLUSION AND FUTURE WORKS 121 5. 123 Bibliography 125 PUBLICATIONS 139 v ABBREVIATIONS No. Abbreviation Meaning 1 API Application Programming Interface 2 CNN Convolution Neural Network 2 CPU Central Processing Unit 3 CVFH Clustered Viewpoint Feature Histogram 4 FN False Negative 5 FP False Positive 6 FPFH Fast Point Feature Histogram 7 fps f rame per second 8 GCSAC Geometrical Constraint SAmple Consensus 9 GPS Global Positioning System 10 GT Ground Truth 11 HT Hough Transform 12 ICP Iterative Closest Point 13 ISS Intrinsic Shape Signatures 14 JI Jaccard Index 15 KDES Kernel DEScriptors 16 KNN K Nearest Neighbors 17 LBP Local Binary Patterns 18 LMNN Large Margin Nearest Neighbor 19 LMS Least Mean of Squares 20 LO-RANSAC Locally Optimized RANSAC 21 LRF Local Receptive Fields 22 LSM Least Squares Method 23 MAPSAC Maximum A Posteriori SAmple Consensus 24 MLESAC Maximum Likelihood Estimation SAmple Consensus 25 MS MicroSoft 26 MSAC M-estimator SAmple Consensus 27 MSI Modified Plessey 28 MSS Minimal Sample Set 29 NAPSAC N-Adjacent Points SAmple Consensus vi 30 NARF Normal Aligned Radial Features 31 NN Nearest Neighbor 32 NNDR Nearest Neighbor Distance Ratio 33 OCR Optical Character Recognition 34 OPENCV OPEN source Computer Vision Library 35 PC Persional Computer 36 PCA Principal Component Analysis 37 PCL Point Cloud Library 38 PROSAC PROgressive SAmple Consensus 39 QR code Quick Response Code 40 RAM Random Acess Memory 41 RANSAC RANdom SAmple Consensus 42 RFID Radio-Frequency IDentification 43 R-RANSAC Recursive RANdom SAmple Consensus 44 SDK Software Development Kit 45 SHOT Signature of Histograms of OrienTations 46 SIFT Scale-Invariant Feature Transform 47 SQ SuperQuadric 48 SURF Speeded Up Robust Features 49 SVM Support Vector Machine 50 TN True Negative 51 TP True Positive 52 TTS Text To Speech 53 UPC Universal Product Code 54 URL Uniform Resource Locator 55 USAC A Universal Framework for Random SAmple Consensus 56 VFH Viewpoint Feature Histogram 57 VIP Visually Impaired Person 57 VIPs Visually Impaired People vii LIST OF TABLES Table 2.1 The number of frames of each scene.2 The average result of detected table plane on our own dataset(%).3 The average result of detected table plane on the dataset [117] (%).4 The average result of detected table plane of our method with different down sampling factors on our dataset.1 The characteristics of the generated cylinder, sphere, cone dataset (synthesized dataset) .2 The average evaluation results of synthesized datasets.
The syn- thesized datasets were repeated 50 times for statistically representative results.3 Experimental results on the ’second cylinder’ dataset. The exper- iments were repeated 20 times, then errors are averaged.4 The average evaluation results on the ’second sphere’, ’second cone’ datasets. The real datasets were repeated 20 times for statistically representative results.5 Average results of the evaluation measurements using GCSAC and MLESAC on three datasets. The fitting procedures were repeated 50 times for statistical evaluations.1 The average result detecting spherical objects on two stages.2 The average results of detecting the cylindrical objects at the first stage in both the first and second datasets.3 The average results of detecting the cylindrical objects at the second stage in both the first and second datasets.4 The average processing time of detecting cylindrical objects in both the first and second datasets.5 The average results of 3-D queried objects detection.
116 viii LIST OF FIGURES Figure 1 Illustration of a real scenario: a VIP comes to the Kitchen and gives a query: ”Where is a coffee cup? ” on the table. Left panel shows a Kinect mounted on the human’s chest. Right panel: the developed system is build on a Laptop PC. 2 Figure 2 Illustration of the process of 3-D query-based object in the indoor environment.
The full object model is the estimated green cylinder from the point cloud of coffee-cup (red points). 3 Figure 3 A general framework of detecting the 3-D queried objects on the table of the VIPs.1 Illustration of the 3-D object recognition process towards local feature based method [53].2 Illustration of primitive shapes extraction from the point cloud [144] .3 Illustration of the Least squares process.4 Line presentation in image space and in Hough space [126].5 Illustration of line estimation by RANSAC algorithm.6 Diagram of RANSAC-based algorithms.1 Microsoft Kinect Sensor version 1.2 Illustration of the organized point cloud representation process.3 Description of the organized and unorganized point cloud.4 (a) is a RGB image, (b) is a point cloud of a scene.5 The proposed framework for table plane detection.6 (a) Computing the depth value of the center pixel based on its neighborhoods (within a (3 × 3) pixels window); (b) down sampling of the depth image.7 Illustration of estimating the normal vector of a set point in the 3-D space. (a) a set of points; (b) estimation of the normal vector of a black point; (c) selection of two points for estimating a plane; (d) the normal vector of a black point.8 Illustration of point cloud segmentation process.9 Example of plane segmentation (a) color image of the scene; (b) plane segmentation result with PROSAC in a our publication; (c) plane segmentation result with the organized point cloud.10 Illustrating acceleration vector provided by a Microsoft Kinect sensor. (xk , yk , zk ) are the three coordinate axes of the Kinect coordinate system that mounted on the chest of VIP.11 Illustration of extracting the table plane in the complex scene.12 Examples of 10 scenes captured in our dataset.13 Scenes in the dataset [?].14 (a) Color and depth image of the scene; (b) mask data of table plane; (c) cropped region; (d) point cloud corresponding of the cropped region, green point is 3-D centroid of the region.15 (a), Illustration of the angle between normal vector of the de- tected table plane and T.
