VIETNAM NATIONAL UNIVERSITY HOCHIMINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY ADVANCED PROGRAM IN INFORMATION SYSTEMS LE DANG XUAN THUY RESEARCH FACIAL MOTION PRIOR NETWORK MODEL AND APPLY TO MOOD TRACKING DIARY BACHELOR OF ENGINEERING IN INFORMATION SYSTEMS HO CHI MINH CITY, 2021 NATIONAL UNIVERSITY HOCHIMINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY ADVANCED PROGRAM IN INFORMATION SYSTEMS LE DANG XUAN THUY - 17521120 RESEARCH FACIAL MOTION PRIOR NETWORK MODEL AND APPLY TO MOOD TRACKING DIARY BACHELOR OF ENGINEERING IN INFORMATION SYSTEMS THESIS ADVISOR PhD. CAO THI NHAN HO CHI MINH CITY, 2021 ASSESSMENT COMMITTEE The Assessment Committee is established under the Decision. by Rector of the University of Information Technology. ACKNOWLEDGMENTS For this graduate thesis, I would like to express my sincere gratitude to my supervisor, PhD.
Cao Thi Nhan took her precious time to guide and pass on her experiences to me. I feel very fortunate to have had the opportunity to enrich my knowledge with her, her guidance, encouragement, and suggestions have provided me with much-needed insight into this thesis. Without her support and advice, I probably wouldn't have done my best job. At the same time, I also want to send to the teachers and teaching assistants in the University of Information Technology, especially the teachers in the Faculty of Information Systems, who have imparted valuable knowledge to me during these 4 years here.
I also want to thank my brothers and sisters, friends who have always been there to help and encourage me whenI faced difficulties and felt like falling. I consider this an important milestone in my education and development. I will try to use the skills and knowledge gained in the best possible way and I will continue to improve more in the future. Once again, I would like to sincerely thank everyone for their help over the years.
Sincerely, Le Dang Xuan Thuy ACKNOWLEDGMENTS TABLE OF CONTENTS. LIST OF FIGURES .--«s«<<es<«es«« LIST OF TABLES. LIST OF ACRONYMS AND ABBREVIATIONS. ABSTRAC Chapter 1 PROBLEM STATEMENT 1.2 Aims and Obj€CfV€S.
6c Sàn TH TH Hành 1 1. 1⁄4 Structure of th Chapter 2 THEORETICAL BACKGROUND AND LITERATURE REVIEW.3 Face Expression Recognition Proces: 2.5 Convolutional Neural Network —CNN: 2.1 Related works with the traditional metho: 2.2 Related work with deep learning method.1 Facial Motion Prior Network Model.1 Facial-Motion Mask Generator (FMG) 3. Prior Fusion Net (PFN).2 Training model FMPN 3.2 Set up the environment for model. Run model with CKPlus dataset 3.4 Run model with JAFFE dataset.2 JAFFE Chapter 4 IMPLEMENTATION MOOD TRACKING DIARY.-ccccccccc trệt rrrrrrrgrerug 4.2 Scope of appli 4.
System analyze design 4.1 Use case diagram .2 Entity Relationship Diagram (ERD). Write diary process 4.4 Development application for mood1d EM ¬. LIST OF FIGURES caso Figure 2.1 Different techniques for face deteCtiOn.2 Six basic categories of facial expression [5].3 Process of facial expression recognition.4 Relation of Artificial intelligence, Machine learning and Deep learning from Oracle oo.5 Model of Convolutional Neural Network from the general structure .6 Timeline of CNNs from Illustrated: 10 CNN Architectures website .10 Facial expression recognition system of LBP by Ekweariri et.11 Face detection by divided into 20 blocks by Ekweariri et.12 Label image for classes of LBP by Ekweariri et.13 Feature extraction by LBP of Ekweariri et.14 Graph of the facial expression recognition model of Li et.15 Face detection by first CNNs by FERC of Ninad Mehendale.16 Facial expression vector detection by second CNNs by FERC of Ninad Mehendale .1 Architecture of the proposed method FMPN Figure 3.2 Description of FMG stage .3 Ground truth mask of the CK+ dataset. Corresponding expressions from left to right, top to bottom are anger, contempt, disgust, fear, happiness, sadness and surprise.4 Description of PFN stage Figure 3.5 Description of CN stage.8 Tracking the learning and loss rate of creating res_face in the visdom server.9 Tracking the learning and loss rate of fusion face in the visdom SCTVT.
HT HH TH TL HH H10 H101 101011110 28 Figure 3.10 Result for testing folder Ö.11 Tracking the learning and loss rate of creating res_face in the ViSUOM SELVEL.12 Tracking the learning and loss rate of fusion face in the visdom Figure 3.13 Result for testing folder 0.14 Confusion matrix in thesis training (left) and author’s (right) [10] Figure 3.15 Contempt emotion image has wrong predicted happy label in thesis 35 Figure 3.16 Confusion matrix on the JAFFE in thesis.17 Sadness emotion image has wrong predicted anger label in thesis .1 Use case diagram for system WebApp Emotion Diary Figure 4.2 Database Overview for Emotion Diary Figure 4.3 Create the database with User, Diary, Emotion table in SQLite.4 User table visualization by DB Browser for SQLite.5 Some of the values from account users registered in the web .6 Some values of user’s post diaries .7 Values of emotion Figure 4.8 Process of writing a diary in web application 45 Figure 4.9 Screen of the login account Figure 4.10 Screen of the registered aCCOUNE .11 Screen for user infOrmatiOI.12 Screen for ask open CaI€TA.13 Screen for start CAIT€TA. + St EkeEeEkEkEkekrrkrkerrrkrkrkrree Figure 4 4.14 Screen for taking the picture.15 Screen for predict emotion and write a diary.16 Screen for fetch diary after choosing in the calendar to update.17 Screen of chart counts the number of emotions in a diary.18 Screen of chart track the streak of the user for giving advice. 48 LIST OF TABLES caso Table 2-2 Summary of recognition tests of Ekweariri ef.- - - -+-«ex+<<+ Table 2-3 Action units related to facial expressions mentioned by Li et.al Table 2-4 Summary of recognition test of Li et. eee Table 3-1 Comparison FMPN and FERC .-- --5- 5-5255 S++xs£xerrxerxerxer Table 3-2 Label of €XDT€SSIOH.
-- SE kg HT rrdnrư 25 Table 3-3 Average accuracy result testing using k-cross validation with 10 folders in thesis Table 3-4 Average accuracies on CKPlus in FMPN of Yudong Chen et. 32 Table 3-5 Classification report of the CKPlus in thesis.--------:-<-- 34 Table 3-6 Result testing using k-cross validation with 10 folders in thesis .35 Table 3-7 Classification report on the JAFFE in thesis .-------:---- 36 Table 4-1 The population of some applications related to emotion diary. 39 Table 4-2 The population of some applications related to diary. Table 4-3 The population of some applications related to mental health care.
Table 4-4 Use case description of writing diary Table 4-5 Use case description of emotion detection. --:-‹--s--s-5<5s+5s++ 42 vi LIST OF ACRONYMS AND ABBREVIATIONS Meaning FER Facial Expression Recognition FMPN Facial Motion Prior Networks FMG Facial-Motion Mask Generator PEN Prior Fusion Network CN Classification Net CNN Convolutional Neural Network FACs Facial Action Coding System EFACs Emotion Facial Action Coding System VGG Visual Geometry Group ResNet Residual Network GPUs Graphics Processing Unit SVM Support Vector Machine LBP Local Binary Pattern SAANet Siamese Action-units Attention Network DBN Deep Belief Network Architechture KNN K-nearest neighbor DCT Discrete Cosine Transform vii 18 HOG Histogram of Oriented Gradients 19 MLP Multi Layer Perceptron 20 VTB Vector Time Backward 21 dLHD Directed Line Segment Hausdorff Distance 22 LEM Line Segment Hausdorff 23 GANs Generative Adversarial Networks 24 CK+ The Extended Cohn-Kanade Dataset 25 JAFFE The Japanese Female Facial Expression Dataset Vii ABSTRACT Derived from facial recognition, facial emotion recognition is also more interested in the field of computer vision and image processing. This problem will receive input as an image of the face that is showing nuances: happy, sad, angry,. And the output will be one of the corresponding emotional labels.
In the period from the 2000s onwards, facial recognition in general and the application of facial recognition to read expressions, in particular, received a lot of attention. When I researched and surveyed the articles in the last 5 years, I found quite a few related articles. With the current age of development, the problem of facial emotion recognition is increasingly focused on developing because of its application in reality. With a system capable of reading facial emotions, we can apply this system to medical facilities, health care centres, customer care as well as some sectors in the industry such as cinematography or legal facilities to be able to make small reminders, alert, supervise and support work in social and community work.
In this graduate thesis, I focused on researching and learning about the problem of facial emotion recognition according to the FMPN method. Along with that, I take advantage of the existing knowledge along with learning more knowledge about machine learning, deep learning to apply the model and build an application for the above problem. ix Chapter 1 PROBLEM STATEMENT 1.1 Rationale The face is the first place we look at to identify someone in the crowd. That is for facial recognition, facial emotion recognition is reading and analyzing, recognizing the difference between the expressive face and the normal face without expression.
From there, we know what the other person's feelings are. Just as reading a composition or a poem means a deep understanding of the writer's meaning, the emotion on the face also denotes many layers of human thought and attitude. It's easy for people to read each other's feelings, but it's a lot more difficult for the machine to read. Therefore, the study of this problem is necessary, not only the topic of my application but also in many other areas of society.
And the important question is the accuracy of the emotion as well as the ability to predict emotions in its depth. This is evidenced by the numerous research articles related to this issue and more and more model-building methods to improve the reliability and speed of facial processing in general and facial expressions in particular. In my speciality is Information Systems, this problem has a certain relevance. In the system definition, the application after being completed can be applied integrated into care systems in key areas, such as health, business.
In the definition of information, a large amount of information coming from emotions is huge because a person has a lot of emotional aspects, each emotion has a variety of levels of expression and at the same time, there are cases where the information received when predicting the emotions and the actual information that the person provides expresses may be different because the emotion is false, not the real emotion.2 Aims and Objectives With this graduate thesis, the main goal that I aim for when doing this is as follows: e Learning about facial recognition, how to apply deep learning to solve that problem. e Understanding the problem of recognizing facial emotions. Survey several relevant research-oriented articles and current applications in the community. e Understanding and applying the model to solve the problem of facial emotion recognition, thereby building an application to support monitoring and reminders for mental and emotional health.3 Scope of study This course focuses on the study of the Facial Motion Prior Network method.
This is considered a FER Framework, to recognize facial emotions. This FMPN method applies the knowledge of Deep Learning and Prior knowledge to create a facial mask construction machine learning model. And as the name implies, this method focuses on moving facial muscles instead of the differences between landmarks on facial parts. With the application of this FMPN method to the practical application of emotional journaling applications, minors are essential.
The main reason is the need to balance the care of the body health, and mental health also needs to be taken care of. Therefore, this app is built to support teens to monitor emotions and moods in the form of logs. And the app aims to make the mental health care of users accessible in hospitals that monitor patients.4 Structure of thesis The thesis report is organized as follow: Chapter 2 is general knowledge with theoretical background and literature review. Chapter 3 shows the experiment and result of the facial motion prior network model.
Chapter 4 presents the design and development of the demo web application. In Chapter 5, the conclusion after the study about facial expression recognition with deep learning and future work will be shown.