MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT AUTOMATION AND CONTROL ENGINEERING TECHNOLOGY SERVICE ROBOT FOR STUDENTS BASED ON COMPUTER VISION AND NATURAL LANGUAGE PROCESSING LECTURER: ASSOC. LE MY HA STUDENT: NGUYEN TUAN THANH SKL009325 Ho Chi Minh City, August, 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT SERVICE ROBOT FOR STUDENTS BASED ON COMPUTER VISION AND NATURAL LANGUAGE PROCESSING NGUYỄN TUẤN THANH Student ID: 17151028 Major: AUTOMATION AND CONTROL ENGINEERING TECHNOLOGY Advisor: Assoc. LE MY HA Ho Chi Minh City, August 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT SERVICE ROBOT FOR STUDENTS BASED ON COMPUTER VISION AND NATURAL LANGUAGE PROCESSING NGUYỄN TUẤN THANH Student ID: 17151028 Major: AUTOMATION AND CONTROL ENGINEERING TECHNOLOGY Advisor: Assoc. LE MY HA Ho Chi Minh City, August 2022 Faculty for High Quality Training – HCMC University of Technology and Education THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, August 6th, 2022 GRADUATION PROJECT ASSIGNMENT Student name: Nguyen Tuan Thanh Student ID: 17151028 Major: Automation and Control Engineering Class: 17151CLA1 Technology Advisor: Assoc.
Le My Ha Phone number: 0938811201 Date of assignment: Feb 21th, 2022 Date of submission: August 6th, 2022 1. Project title: Service robot for students based on computer vision and natural language processing. Initial materials provided by the advisor: References, reference programs, data sets, expected parameters of the Robot. Content of the project: - Design, implement a service robot with two functions: chat and talk.
- Apply computer vision to identify wearing a mask and user information. - Apply natural language processing in virtual voice assistant to communicate with human. - Apply natural language toolkit (NLTK) to build chatbot to communicate with human. - Build database and collect more database when communicate with human.
Final product: Finish a service robot that have abilities to recognize human with high accuracy and communicating with human by given knowledge database. CHAIR OF THE PROGRAM ADVISOR (Sign with full name) (Sign with full name) Faculty for High Quality Training – HCMC University of Technology and Education THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, August 6, 2022 ADVISOR’S EVALUATION SHEET Student name: Nguyen Tuan Thanh Student ID: 17151028 Major: Automation and Control Engineering Technology Project title: Service robot for students based on computer vision and natural language processing Advisor: Assoc. Le My Ha EVALUATION 1. Content of the project: - Design, implement a service robot with two functions: chat and talk.
- Apply computer vision to identify wearing a mask and user information. - Apply natural language processing in virtual voice assistant to communicate with human. - Apply natural language toolkit (NLTK) to build chat bot to communicate with human. - Build database and collect more database when communicate with human.
Approval for oral defense? (Approved or denied) .) Ho Chi Minh City, August 6th, 2022 ADVISOR (Sign with full name) Faculty for High Quality Training – HCMC University of Technology and Education THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, August 6, 2022 PRE-DEFENSE EVALUATION SHEET Student name: Nguyen Tuan Thanh Student ID: 17151028 Major: Automation and Control Engineering Technology Project title: Service robot for students based on computer vision and natural language processing Name of Reviewer:. Content of the project: - Design, implement a service robot with two functions: chat and talk. - Apply computer vision to identify wearing a mask and user information. - Apply natural language processing in virtual voice assistant to communicate with human.
- Apply natural language toolkit (NLTK) to build chat bot to communicate with human. - Build database and collect more database when communicate with human. Approval for oral defense? (Approved or denied) .) Ho Chi Minh City, August 6th, 2022 REVIEWER (Sign with full name) Faculty for High Quality Training – HCMC University of Technology and Education THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name: Nguyen Tuan Thanh Student ID: 17151028 Major: Automation and Control Engineering Technology Project title: Service robot for students based on computer vision and natural language processing Name of Defense Committee Member:. Content of the project: - Design, implement a service robot with two functions: chat and talk.
- Apply computer vision to identify wearing a mask and user information. - Apply natural language processing in virtual voice assistant to communicate with human. - Apply natural language toolkit (NLTK) to build chat bot to communicate with human. - Build database and collect more database when communicate with human.) Ho Chi Minh City, August 6th, 2022 COMMITTEE MEMBER (Sign with full name) Graduation Thesis ACKNOWLEDGEMENT In the process of completing the graduation project, in addition to my own understanding, I have received a lot of support and dedicated help.
First, I would like to express my deep gratitude to Associate Professor Dr. Le My Ha, who is both a teacher, a supporter, and an inspiration for me to complete this thesis. He oriented me to the right topic, and how to do it, and gave objective feedback to help me when defending myself in front of the council. Therefore, I feel very fortunate to have worked with him.
Next, I would like to thank the faculty of electronics and electronics faculty as well as the high-quality training department for imparting useful knowledge during four years at the university. This knowledge plays a fundamental role in the implementation of my graduation thesis. In addition, I would also like to thank the Intelligent Systems Laboratory (ISLAB) of the Faculty of Electrical and Electronic Engineering for supporting me in terms of facilities as well as useful knowledge during the completion of the project. And indispensable is the deep thanks to a friend Tran Thanh Hung who supported and guided me to develop this topic.
Finally, I would like to thank my family for always supporting, caring, and motivating me to complete the project in the best possible way. Ho Chi Minh city, August 6th 2022 Student Graduation Thesis Table of Contents CHAPTER 1: INTRODUCTION .5 Approach and research. 2 CHAPTER 2: LITERATURE REVIEW .1 Survey of robots being used in service industry .1 Mission of robots in the service industry .2 Background of face recognition system .2 Structure and procedure for face recognition .3 Color spaces in image processing .1 RGB color space (Red-Green-Blue) .2 HSV color space (Hue-Saturation-Value) .4 Histogram of Oriented Gradients algorithm .5 Support Vector Machine algorithm .6 Background of speech recognition system .7 Framework and libraries. 19 Graduation Thesis CHAPTER 3: SYSTEM DESIGN AND CONSTRUCTION .1 Requirements of the system .1 The block diagram of the system .2 The function of each block .2 Face recognition and identification: .3 Face mask detection .4 Speech recognition and voice assistant .30 CHAPTER 4: EXPERIMENT RESULTS, FINDINGS AND ANALYSIS .2 Face recognition and identification .1 Training image data .2 Performing the face recognition .3 Face mask detection .4 Speech recognition and voice assistant .1 Create Training Data.
46 CHAPTER 5: CONCLUSIONS AND DIRECTIONS OF DEVELOPMENT .2 Direction of development .48 Graduation Thesis ABBREVIATIONS NLP: Natural Language Processing OpenCV: Open Source Computer Vision Library HOG: Histogram of Oriented Gradients SVM: Support Vector Machine Q&A: Question and answer Graduation Thesis List of figures Figure 2. Pepper robot working in a mobile store. A typically procedure for face recognition model. Face and eye detection.
RGB color space (Red-Green-Blue). HSV color space (Hue-Saturation-Value). Applications of HOG. An example of support vector in 2-Dimensional data.
Margins describing in a plane. An example of linearly non separable dataset. Implementation of Speech Recognition. Interface of Windows Speech Recognition.
Interface of Voice-To-Text Facebook Messenger. Interface of Google Speech to Text. Pytorch and TensorFlow Frameworks from 2017 to 2021 [10]. Reading CSV files with Pandas [11].
Example illustrating some functions in Numpy. Market share of voice assistants in the US, May 2018 [12]. Illustration for Chatbot. Block diagram of service robot designing by student.
Face detection with 6 landmarks and multi-face support [17]. Training Process of face recognition. Five features of Haar cascade method [18]. (c) Four-rectangle feature.
Cascade structure for Haar classifiers [18]. Sliding window in grayscale image [19]. Image meshing and histogram calculation [19]. Face recognition and identification processing.
Face mask detection process .29 Graduation Thesis Figure 3. Plotting all the milestone central issues of an individual's face on a white foundation can provide us with a best guess of the shape [20]. The structure of training data [22]. Example of training data bag of words [22].
Example of NLP preprocessing pipeline [22]. Structure of Feed Forward Neural Network [23]. The simplest form of perceptron [23]. Chatbot training structure [22].
Six facial features are displayed when human face is detected and frame rate is measured. Detecting multiple faces in the same frame. The process of training image data. Username recognition and display.
Detect 68 landmarks on user's face. Bounding the mouth and warning when the user is not wearing a mask. When the user wears a mask, the system will not give an alert. Identify and answer questions from users when the question is in the data set.
Identify and answer questions from users when the question is not in the data set. Save unknown questions to unknown question sheet in excel. Relative calculation of response speed of gtts library. Relative calculation of response speed of pyttsx3 library.
Training data made by the student. Tokenize all questions from data file. Lowercase all word tokenized and remove characters. All words after remove duplicate word and sorted.
Example of the bag of words for all patterns. User interface designed by the student. 46 Graduation Thesis List of Tables Table 2- 1 Specifications of Pepper robot. 5 Table 2- 2 The speech recognition package in Python.
15 Table 4- 1 Sample collects data from students .40 Graduation Thesis ABSTRACT With the advancement of science and technology, robots are gradually replacing humans in work or help in daily life. Similarly, to bring convenience to answering the daily questions of students, this project will design a service robot that combines computer vision and natural language processing to adapt to this purpose. Compared to the traditional way of answering questions, students can go to school personnel or message student forums to ask about the problem they are facing. These forms will often take a lot of time because the response time is often quite long, the number of staff is limited, and the number of students asking questions is often quite large.
Therefore, this topic proposes a solution to replace traditional question-answering forms with robots capable of consulting and answering questions of students through two forms of communication: talking and chatting. When in talking mode, the robot will recognize the user, recognize the question by voice, and process it to give the appropriate answer. When in chatting mode, the user will enter a question into the chat box, and then the robot will process and give an appropriate answer. From the descriptions above, this topic shows the convenience of answering questions of students quickly, saving human resources for the school, and at the same time capturing objectively questions of students.
Keywords: service robot, computer vision, natural language processing. Graduation Thesis Chapter 1: Introduction CHAPTER 1: INTRODUCTION 1.1 Define a problem In the field of education, in addition to imparting useful knowledge, it is also necessary to listen to and answer the questions of students in the most effective way.