MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY TAVN NVA NVOH Hoang Van Nam AONATOS WALAAAOD DIFFICULT SITUATIONS RECOGNITION SYSTEM FOR VISUALLY-IMPAIRED AID USING A MOBILE KINECT MASTER THESIS OF SCTENCE COMPUTER SCIENCE arloz Ha Noi — 2016 MINISTRY OF EDUC. ON AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Hoang Van Nam DIFFICULT SITUATIONS RECOGNITION SYSTEM FOR VISUALLY-IMPAIRED AID USING A MOBILE KINECT Department COMPUTER SCIENCE MASTER THESIS OF SCIENCE SUPERVISOR : 1, Dr. Le Thi Lan Ha Noi 2016 CỘNG TIÒA XÃ TIỘI CHỦ NGIĨA VIỆT NAM Độc lập — Tự đoø — Hạnh phúc BẢN XÁC NHẬN CHỈNH SỬA LUẬN VĂN THẠC SĨ Họ và tên tác giả luận văn : Để tài luận văn: Chuyên ngành: Mã số SV Tác giả, Người hướng dẫn khoa học và Hội dồng chấm luận văn xác nhận tác giả đã sửa chữa, bố sung luận văn theo biên bản họp Hội đồng ngày - với các nội dung sau. Ngày tháng năm Giản viên hưởng dẫn Tác giá luận văn CTITỦ TỊCTI HỘI ĐÒNG Contents v 9:6: O@WGIE ĐEUNGB wo nce os wise nies cà tein eee ne ee ni ae 32 3⁄7 Stair Detection 34 3 Stair definition.
34 3, Color-based stair dete, 35 3.3 Depth-based stair detection „ đỗ 3.8 Obstacle information representation 48 4 Experiments 49 41 Đế vỉ ví ví cà: 49 4.2 Difficult situation recognition evaluation 51 4.1 Obstacle detection evaluation 51 4.2 Stair detecion evaluation. cà co cà cà 53 5 Conclusions and Future Works 58 Bed ĐGGUNNEB ¿vị se ene ase se RA eGR SEW RGR Ee RF RR A 58 5.2 Future Works oe ee 59 6 Publications 60 Bibliography 61 HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Abstract International Research Institute MICA Computer Vision Department Master of Science Difficult situations recognition for visual-impaired aid using mobile Kinect by Hoang Van NAM By 2014, according to figures from some organization, here are more than one million people in the Vietnam living with sight loss, about 1.3% of Vietnamese peopl. Although the big impact to the daily living, especially with the ability to move, read, communicate with another, only a small percentage of blind or visually impaired people live with assistive device or animal such as a dog guide. Motivated by the significant changes in technology have take place in the last decade, especially in the introduction of varies typ s of sensors as well as the development in the field of computer vision, I present in this thesis a difficult situations recognition system for visually impaired aid using a mobile Kinect, This system is based on data captured from Kinect and using computer vision technique to detect obstacle.
At the current prototype, I only focused on detecting obstacle in the indoor environment like public building and two types of obstacle will be exploited: general obstacle in the moving way and staircases-which causes a big dangerous to the visually impaired people. The 3D imaging techniques were used to detect the general obstacle including: plane segmentation, 3D point clustering and the mixed strategy between depth and color image is used to detect the staircase based on detecting the stair edges and its structure. The system is very reliable with the detection rate is about 82.9% and the time to process each frame is 493 ms. Acknowledgements Tam eo honor ta he here the second time, in one of the finest imiversity in Vietnam to write those grateful words to people who have been supporting, guiding me from the very first moment when 1 was a university student until now, when I am writing my master thesis.
Lam grateful ta my supervisor, Dr. Le ‘Thi Lan, whose expertise, understanding, gener- ‘ons gnidanee and snpport made it possihie for me to work on a topie that we of great: interest to me. It was a pleasure to work with her. Special thanks to Dr.
Tran Thi Thanh Hai, Dr. Vu Hal and Dr. Nguyên Thì Thuy (VIXUA) and all of the members in the Computer Vision Department, MICA Institute for their sharp comments, guidance for my works which helps me a lot in how to study and to do research in right way and also tne valuable advices and encouragements that Ubey gave lo me during my thesis. T wonld like to express my gratimnde to Prof.
Veelaert: Peter, Dr. Taeng, Quang Mep and Mr. Michiel Vlaminck at Ghent University, Belgium for their supporting. It’s been a great honor to cooperate and work with them.
‘inaliy, 1 wouid especially like to thank my family and friends jor their continues love. support they have given me throng my life, helps me pass through all the frustrating, struggling. ‘I'hanks for everything that helped me get to this day. Hanoi, 19/02/2016 Hoang Van Nam iit Acknowledgements Tam eo honor ta he here the second time, in one of the finest imiversity in Vietnam to write those grateful words to people who have been supporting, guiding me from the very first moment when 1 was a university student until now, when I am writing my master thesis.
Lam grateful ta my supervisor, Dr. Le ‘Thi Lan, whose expertise, understanding, gener- ‘ons gnidanee and snpport made it possihie for me to work on a topie that we of great: interest to me. It was a pleasure to work with her. Special thanks to Dr.
Tran Thi Thanh Hai, Dr. Vu Hal and Dr. Nguyên Thì Thuy (VIXUA) and all of the members in the Computer Vision Department, MICA Institute for their sharp comments, guidance for my works which helps me a lot in how to study and to do research in right way and also tne valuable advices and encouragements that Ubey gave lo me during my thesis. T wonld like to express my gratimnde to Prof.
Veelaert: Peter, Dr. Taeng, Quang Mep and Mr. Michiel Vlaminck at Ghent University, Belgium for their supporting. It’s been a great honor to cooperate and work with them.
‘inaliy, 1 wouid especially like to thank my family and friends jor their continues love. support they have given me throng my life, helps me pass through all the frustrating, struggling. ‘I'hanks for everything that helped me get to this day. Hanoi, 19/02/2016 Hoang Van Nam iit Contents Declaration of Authorship Abstract Acknowledgements Contents List of Figures List of Tables Abbreviations 1 Introduction 11 Motivation wee 1.2 Difficult situations ae Mobile Kineet .24 Environment Context AL Difficult Situations Recognition 12 1.4 Thesis Contributions 13 2 Related Works 14 21 As systems for visually impaired people.
14 RGB-D based assistive systems for visually impaired people “ 18 8) SUURDASRON coe ws one ware nese eae conte are wt ne te 20018 trà E 19 3 Obstacle Detection 25 31 Oveview. 36 3⁄8 Point Clond Registration 27 34 Plane Segmentation 30 3.5 Ground & Wall Plane Dete 32 iv Declaration of Authorship I, Hoang Van NAM, declare Unal chis Uhesis viuled, ‘Difficult situations recognition for visualimpaired aid usiag mobile Kinect' aud Ue wock preseuted in it are my own. I confirm that: a This work was done wholly or mainly while in candidature for a research degree at this University. = Where auy park of this thesis has previously born subunitied for a degree or any other qualifi ion at thie Uni preity or any other institution, this has been clearly stated = Where T have consulted the published wark of athers.
this is always clearly at- tributed. m Where | have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely my own work. m L have acknowledged all main sources of help.
m Where the thesis is based on work done by myself jointiy with others, | have made clear exactly what was done by others and what | have contributed myself. Acknowledgements Tam eo honor ta he here the second time, in one of the finest imiversity in Vietnam to write those grateful words to people who have been supporting, guiding me from the very first moment when 1 was a university student until now, when I am writing my master thesis. Lam grateful ta my supervisor, Dr. Le ‘Thi Lan, whose expertise, understanding, gener- ‘ons gnidanee and snpport made it possihie for me to work on a topie that we of great: interest to me.
It was a pleasure to work with her. Special thanks to Dr. Tran Thi Thanh Hai, Dr. Vu Hal and Dr.
Nguyên Thì Thuy (VIXUA) and all of the members in the Computer Vision Department, MICA Institute for their sharp comments, guidance for my works which helps me a lot in how to study and to do research in right way and also tne valuable advices and encouragements that Ubey gave lo me during my thesis. T wonld like to express my gratimnde to Prof. Veelaert: Peter, Dr. Taeng, Quang Mep and Mr.
Michiel Vlaminck at Ghent University, Belgium for their supporting. It’s been a great honor to cooperate and work with them. ‘inaliy, 1 wouid especially like to thank my family and friends jor their continues love. support they have given me throng my life, helps me pass through all the frustrating, struggling.
‘I'hanks for everything that helped me get to this day. Hanoi, 19/02/2016 Hoang Van Nam iit Acknowledgements Tam eo honor ta he here the second time, in one of the finest imiversity in Vietnam to write those grateful words to people who have been supporting, guiding me from the very first moment when 1 was a university student until now, when I am writing my master thesis. Lam grateful ta my supervisor, Dr. Le ‘Thi Lan, whose expertise, understanding, gener- ‘ons gnidanee and snpport made it possihie for me to work on a topie that we of great: interest to me.
It was a pleasure to work with her. Special thanks to Dr. Tran Thi Thanh Hai, Dr. Vu Hal and Dr.
Nguyên Thì Thuy (VIXUA) and all of the members in the Computer Vision Department, MICA Institute for their sharp comments, guidance for my works which helps me a lot in how to study and to do research in right way and also tne valuable advices and encouragements that Ubey gave lo me during my thesis. T wonld like to express my gratimnde to Prof. Veelaert: Peter, Dr. Taeng, Quang Mep and Mr.
Michiel Vlaminck at Ghent University, Belgium for their supporting. It’s been a great honor to cooperate and work with them. ‘inaliy, 1 wouid especially like to thank my family and friends jor their continues love. support they have given me throng my life, helps me pass through all the frustrating, struggling.
‘I'hanks for everything that helped me get to this day. Hanoi, 19/02/2016 Hoang Van Nam iit List of Figures vii 2.7 A near-approach for stair detection in [13] (A) Input image with detected stair region, (B) Texture energy, (C)Input image with detected lines are stair candidates, (D)Optical flow maps in this image, there is a significant changing in the line in the edge of stair.8 Example of segmentation and classification in [24] 33 2/9 Stair modelins(left) and features in each plane [24].10 Stair detection algorithm proposed in [29] (A) Detected line in the edge image (using color infomation) (B) Depth profiles in each line (red line: pedestrian crosswalk, bhue: down stair, green: upstair) ©.1 Obstacle Detection Flowchart 26 3.2 Kinect mounted on body. x #7 3⁄3 Coordinate Transformation Pro -.98 BA Kinet Coordinate oe ónẽố.5 Point Clond rotation using normal vector of ground plane (while arrow): left: before rotating, right: after rotating 3.6 Normal vector estimation algorithms [15] (a) Normal vector of the center point can be calculated by a eross product of two vectors of four neighbor points (red), (b) Normal vector estimation ina scene.7 Plane segmentation result using algorithm proposed in [15]. Each plane is represented by a distinetiVe eoloF.
