MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION CAPSTONE PROJECT AUTOMATION AND CONTROL ENGINEERING TECHNOLOGY DESIGN AND IMPLEMENT RECEPTION ROBOT FOR STUDENT BASE ON COMPUTER VISION AND NATURAL LANGUAGE PROCESSING LECTURER: Assoc. LE MY HA STUDENT: TRAN THANH HUNG HUYNH XUAN TAN SKL 0 0 9 1 1 9 Ho Chi Minh City, January 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT DESIGN AND IMPLEMENT RECEPTION ROBOT FOR STUDENT BASE ON COMPUTER VISION AND NATURAL LANGUAGE PROCESSING TRAN THANH HUNG HUYNH XUAN TAN Student ID: 17151210 Student ID: 17151027 Major: AUTOMATION AND CONTROL ENGINEERING TECHNOLOGY Advisor: Assoc. LE MY HA Ho Chi Minh City, January 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT DESIGN AND IMPLEMENT RECEPTION ROBOT FOR STUDENT BASE ON COMPUTER VISION AND NATURAL LANGUAGE PROCESSING TRAN THANH HUNG HUYNH XUAN TAN Student ID: 17151210 Student ID: 17151027 Major: AUTOMATION AND CONTROL ENGINEERING TECHNOLOGY Advisor: Assoc. LE MY HA Ho Chi Minh City, January 2022 THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, January 07, 2021 GRADUATION PROJECT ASSIGNMENT Student name: Tran Thanh Hung Student ID: 17151210 Student name: Huynh Xuan Tan Student ID: 17151027 Major: Automation and Control Engineering Class: 17151CLA Technology Advisor: Assoc.
Le My Ha Phone number: Date of assignment: Jan 07th, 2021 Date of submission: December 28th, 2021 1. Project title: Design and Implement Reception Robot for Student base on Computer Vision and Natural Language Processing 2. Initial materials provided by the advisor: References, scientific articles, reference programs, data sets, expected parameters of the Robot. Content of the project: - Design, implement a reception robot with 3D-printting technology.
- Apply computer vision to recognize human facial. - Apply natural language processing in virtual voice assistant to communicate with human. - Build knowledge database to make robot have ability to chatting with human. Final product: Finish a reception robot that have abilities to recognize human with high accuracy and communicating with human by given knowledge database.
CHAIR OF THE PROGRAM ADVISOR (Sign with full name) (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, December 28, 2021 ADVISOR’S EVALUATION SHEET Student name: Tran Thanh Hung Student ID: 17151210 Student name: Huynh Xuan Tan Student ID: 17151027 Major: Automation and Control Engineering Technology Project title: Design and Implement Reception Robot for Student base on Computer Vision and Natural Language Processing Advisor: Assoc. Le My Ha EVALUATION 1. Content of the project: - Design, implement a reception robot with 3D-printting technology. - Apply computer vision to recognize human facial.
- Apply natural language processing in virtual voice assistant to communicate with human. - Build knowledge database to make robot have ability to chatting with human. Strengths: The content of the thesis fully meets the requirements. Students have designed a Reception Robot that have abilities of recognizing objects and performing basic communications.
The results of face recognition and communication are quite good when experimenting on the Reception Robot. Weaknesses: To consider experiments on more samples to have a reference for evaluating the accuracy of the system 4. Approval for oral defense? (Approved or denied) Approved 5. Overall evaluation: (Excellent, Good, Fair, Poor) Excellent 6.5 (in words: nine point five) Ho Chi Minh City, month day, year ADVISOR (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, December 28, 2021 PRE-DEFENSE EVALUATION SHEET Student name: Tran Thanh Hung Student ID: 17151210 Student name: Huynh Xuan Tan Student ID: 17151027 Major: Automation and Control Engineering Technology Project title: Design and Implement Reception Robot for Student base on Computer Vision and Natural Language Processing Name of Reviewer: PhD.
Vu Van Phong EVALUATION 1. Content of the project: - Design, implement a reception robot with 3D-printting technology. - Apply computer vision to recognize human facial. - Apply natural language processing in virtual voice assistant to communicate with human.
- Build knowledge database to make robot have ability to chatting with human. Strengths: The content of the topic is consistent with the Automation and Control Engineering graduation project. The results of the project are good, the workload is huge. Weaknesses: In Section 5.1, face mask detection is presented but there are no test results.
The team needs to supplement the test results and evaluate the accuracy. The use of speech recognition method should be considered to switch the mode from speech processing to image processing or vice versa as the presentation goal of the project is to avoid covid but people's use of push buttons can be harmful. Approval for oral defense? (Approved or denied) Approved 5. Overall evaluation: (Excellent, Good, Fair, Poor) Excellent 6.0 (in words: Nine) Ho Chi Minh City, month day, year REVIEWER (Sign with full name) ACKNOWLEDGEMENT In the process of implementing and completing the graduation project, in addition to our own knowledge, our team has received a lot of support and dedicated help.
First of all, the team would like to send a special thanks to Mr. Le My Ha, who has directly monitored, motivated and guided our team to successfully complete this project. In addition, the valuable knowledge in the process of studying and researching in the courses at the school is an important preparation to carry out this project. Through this, our group would like to sincerely thank the teachers in the electrical and electronic faculty and the high-quality faculty of Ho Chi Minh City University of Technology and Education for teaching important lessons to our group.
At the same time, the team also sincerely thanks the Intelligent Systems Laboratory (ISLAB) of the Faculty of Electrical and Electronic Engineering and the members of the Lab for their support in terms of facilities as well as useful knowledge for the team to practice. The group would also like to thank the friends and companions who have helped the group in the past time. Finally, I would like to thank my family for encouraging, motivating and caring to create favorable conditions for the team to successfully complete this project. Ho Chi Minh city, December 28th 2021 Student i ABSTRACT Industrial Revolution 4.0 created a new era for the industry around the world, where automation is at the forefront, but the work that people have been in charge of seems unlikely.
This replacement has been gradually taken over by the robot with superior efficiency. In particular, one of the major inventions that determine the success of robots today is Computer Vision. Today, Computer Vision is being applied in the most important areas in society such as health, education, and the military. For example, surgical robots have been widely used in large hospitals or military robots with a variety of mission functions.
Realizing the growth and power of Computer Vision, this thesis focuses on researching and developing a robot model that helps students by applying facial recognition and voice recognition technology. This robot aims to assist students in answering relevant questions instead of using human resources for this position. The research in this thesis is done through the following steps: Step 1: Learning about reception robot models that already exist on the market, design idea of hardware and applicable technology on this robot; Step 2: Synthesizing the hardware used in the topic parallel with the design and construction of the robot shell model; Step 3: Conducting research and development of facial recognition system using HOG descriptors (Histogram of Oriented Gradients) and SVM classification (Support vector machine) and Vietnamese language voice assistant system using speech recognition techniques. Step 4: Collect sample data for the facial recognition system and train the robot questioning system.
Step 5: Assemble relevant equipment on the robot shell model and run the test of robot functions. Step 6: Fixing errors arising in the test run and improving unwanted problems. At the end of the research process, the obtained product is a reception robot model capable of identifying people and answering questions within the scope quickly and effectively with the lowest error frequency. To sum up, the above research plays a part in the growing popularity of robots and improving their performance.
Keywords: face detection, face recognition, Haar-like features, HOG descriptor, SVM classification, speech recognition, speech to text, text to speech. ii ABBREVIATIONS NLP: Natural Language Processing OpenCV: Open Source Computer Vision Library HOG: Histogram of Oriented Gradients SVM: Support Vector Machine iii Table of Contents ACKNOWLEDGEMENT. iii Table of Contents. iv List of figures.
vii List of Tables. ix CHAPTER 1: INTRODUCTION. 2 CHAPTER 2: LITERATURE REVIEW .1 Survey of robots being used in service industry.1 Mission of robots in the service industry .2 Background of face recognition system .2 Structure and procedure for face recognition .3 Color spaces in image processing.1 RGB color space (Red-Green-Blue).2 HSV color space (Hue-Saturation-Value) .4 Viola-Jones algorithm .2 Haar-like feature .5 Histogram of Oriented Gradients algorithm .2 Gradient calculation methodology .3 Steps of HOG calculation .6 Support Vector Machine algorithm .2 Building optimization problems for SVM.7 Background of speech recognition system .2 The speech signal and its representation .4 Advantages and disadvantages of speed recognition. 26 CHAPTER 3: HARDWARE DESIGN AND IMPLEMENTATION .1 Requirement analysis for hardware selection .2 Computers being used for computer vision and embedded applications .3 Camera and microphone .6 Design sketch of robot frame.
43 CHAPTER 4: SOFTWARE DESIGN AND ALGORITHMS .1 Programing languages and environment being used for computer vision .2 Tools for coding .3 OpenCV and support packages .2 Method of face recognition .3 Method of voice assistant .4 Process flow diagram. 51 CHAPTER 5: EXPERIMENT RESULTS, FINDINGS AND ANALYSIS .1 Face mask detection .4 Reception robot real-time working. 62 v CHAPTER 6: CONCLUSION AND FUTURE WORKS. 65 vi List of figures Figure 2.1 Pepper robot working in a mobile store .2 A typically procedure for face recognition model .3 RGB color space (Red-Green-Blue) .4 HSV color space (Hue-Saturation-Value) .5 Stages in executing Viola-Jones algorithm .6 Haar Feature descriptor .7 An image containing faces after featuring by Haar .8 An example of integral image .9 Stages of image pre-processing.10 Gradient images in horizontal, vertical directions and combination image .11 Calculating Histogram of Gradients from its direction and magnitude .12 An example of special case in calculating Histogram of Gradients .13 9-bin histograms generating from the image .14 Method of normalizing the histogram of image .15 HOG features labeling in the image result .16 An example of support vector in 2-Dimensional data .17 Margins describing in a plane .18 An example of linearly non separable dataset .19 Two-dimensional space and data points .20 Locating the data points closely to the boundary.21 Process from sounds by speaker transmitting to listener .22 Block diagram of speech recognition systems .23 Spectrogram of a typically sound .24 Interface of Windows Speech Recognition.25 Interface of Voice-To-Text Facebook Messenger .26 Interface of Google Speech to Text .1 Some kind of processor in a real-life project .2 NVIDIA Jetson Nano .3 Pinout of NVIDIA Jetson Nano .4 Various junctions of Jetson Nano board .6 Waveshare 7-inch LCD .9 Body robot design .10 The robot face design .11 The robot head design in Solidworks.12 The design of assembled part of the robot in solidworks .13 Connection diagrams of the electronic components .14 The robot parts are printed individually before being put together .15 The head of the robot after device putting.16 The Reception Robot in full view .2 The working interface of Visual Studio Code .3 OpenCV and Python icon .4 Dlib and Ubuntu icon .5 The face recognition step by step diagram .6 Some sample images for one face after capturing.7 SVM model used in the robot .8 Voice assistant block diagram.9 Operational states diagram of this robot .10 Flowchart of robot’s operation .1 The pre-trained facial landmark detector supported by DLib library .2 Output images with facial feature shown in bounding box .