MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION GRADUATION THESIS MAJOR: MECHANICAL ENGINEERING TECHNOLOGY DESIGNING AN INTERACTIVE SMART MIRROR WITH SENTIMENT ANALYSIS FOR MONITORING EMOTION AND MENTAL COMFORT INSTRUCTOR: BUI HA DUC STUDENT: LE MINH THONG HA GIA KINH NGUYEN PHUC NGUYEN Ho Chi Minh city, July 2024 MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY OF MECHANICAL ENGINEERING GRADUATION THESIS DESIGNING AN INTERACTIVE SMART MIRROR WITH SENTIMENT ANALYSIS FOR MONITORING EMOTION AND MENTAL COMFORT Supervisor: BUI HA DUC, PhD Student: LE MINH THONG Student ID: 19134087 Class: 19134 Student: HA GIA KINH Student ID: 20134016 Class: 20134 Student: NGUYEN PHUC NGUYEN Student ID: 20134017 Class: 20134 Year Of Admission: 2020 - 2024 Ho Chi Minh City, July 2024 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY OF MECHANICAL ENGINEERING DEPARTMENT OF MECHATRONICS GRADUATION THESIS DESIGNING AN INTERACTIVE SMART MIRROR WITH SENTIMENT ANALYSIS FOR MONITORING EMOTION AND MENTAL COMFORT Supervisor: BUI HA DUC, PhD Student: LE MINH THONG Student ID: 19134087 Class: 19134 Student: HA GIA KINH Student ID: 20134016 Class: 20134 Student: NGUYEN PHUC NGUYEN Student ID: 20134017 Class: 20134 Year Of Admission: 2020 - 2024 Ho Chi Minh City, July 2024 TRƯỜNG ĐẠI HỌC SƯ PHẠM KỸ THUẬT TP. HCM CỘNG HOÀ XÃ HỘI CHỦ NGHĨA VIỆT NAM KHOA CƠ KHÍ CHẾ TẠO MÁY Độc lập - Tự do – Hạnh phúc NHIỆM VỤ ĐỒ ÁN TỐT NGHIỆP Học kỳ II/ năm học 2023-2024 Giảng viên hướng dẫn: TS. Bùi Hà Đức…………….……… Sinh viên thực hiện: Lê Minh Thông………….MSSV: 19134087…Điện thoại 0387109318 Hà Gia Kính……………….MSSV: 20134016…Điện thoại 0947559196 Nguyễn Phúc Nguyên………MSSV: 20134017…Điện thoại 0704868339 1. Mã số đề tài: CDT5 - Tên đề tài: Designing An Interactive Smart Mirror With Sentiment Analysis For Monitoring Emotion And Mental Comfort 2.
Các số liệu, tài liệu ban đầu: ……………. Nội dung chính của đồ án: ……………. Các sản phẩm dự kiến ……………. Ngôn ngữ trình bày: Bản báo cáo: Tiếng Anh Tiếng Việt Trình bày bảo vệ: Tiếng Anh Tiếng Việt TRƯỞNG KHOA TRƯỞNG BỘ MÔN GIẢNG VIÊN HƯỚNG DẪN (Ký, ghi rõ họ tên) (Ký, ghi rõ họ tên) (Ký, ghi rõ họ tên) i COMMITMENT • Project: Designing An Interactive Smart Mirror With Sentiment Analysis For Monitoring Emotion And Mental Comfort • Lecturer: Bui Ha Duc, Ph.D • Student: Le Minh Thong • Student ID: 19134087 - Class: 19134 • Address: 79/3A Phu Tho Hoa Street, Phu Tho Hoa Ward, Tan Phu District, Ho Chi Minh City • Phone number: 0387109318 • Email: thong.com • Student: Ha Gia Kinh • Student ID: 20134016 - Class: 20134 • Adress: 296 Phu Tho Hoa Street, Phu Tho Hoa Ward, Tan Phu District, Ho Chi Minh City • Phone number: 0947559196 • Email: hagiakinh0209@gmail.com • Student: Nguyen Phuc Nguyen • Student ID: 20134017 - Class: 20134 • Address: Phuoc Thien Commune, Nhon Trach District, Dong Nai Province • Phone number: 0704868339 • Email: phucnguyen382002@gmail.com • Graduation thesis submission date: July …, 2024 • Commitment: “I affirm that the graduate thesis presented here is the result of my research and efforts.
I have not replicated any content from published articles without appropriate citations. Should any breach be identified, I acknowledge full accountability for the consequences.” Ho Chi Minh City, … July 2024 ii ACKNOWLEDGMENT First and foremost, on behalf of my team, I would like to extend my heartfelt thanks to our supervisor, Bui Ha Duc Ph. Your unwavering commitment, expertise, and guidance have been instrumental in shaping my research and steering me towards success. Your mentorship, patience, and constant encouragement have played a significant role in shaping my academic journey.
Your profound knowledge and invaluable insights have helped me overcome challenges and broaden my intellectual horizons. I am truly grateful for your dedication and support. I would also like to extend my sincere appreciation to Ho Chi Minh City University of Technology and Education for providing me with an exceptional learning environment and resources. The university has been a constant source of inspiration, fostering an atmosphere of growth, innovation, and academic excellence.
The diverse faculty and staff have been instrumental in shaping my academic and personal development, and I am indebted to them for their commitment to nurturing future scholars and leaders. My deepest gratitude goes to my family for their unwavering love, encouragement, and understanding. Your support has been the foundation of my journey, and I am forever grateful for your sacrifices, belief in me, and the countless ways you have cheered me on. Your unwavering support has provided me with the strength and motivation to overcome challenges and strive for excellence.
Lastly, I would like to express my gratitude to all those who have contributed to my growth and development, directly or indirectly. Your encouragement, advice, and belief in my abilities havebeen invaluable throughout this challenging yet rewarding journey. Sincerely, Nguyen Phuc Nguyen iii ABSTRACT The rise of the Internet of Things (IOT) has transformed everyday objects into intelligent devices seamlessly integrated into our lives. Smart mirrors, a novel application of this technology, present a unique opportunity to enhance our daily routines.
This thesis explores the design and development of a smart mirror prototype, focusing on its functionalities and potential benefits for the modern user. The thesis begins with an overview of the existing literature on smart mirrors, analyzing their current capabilities and limitations. It identifies key areas for improvement, particularly in user experience and feature integration. The methodology section details the development process, outlining the hardware and software components employed to create the smart mirror prototype.
This includes the selection of display technology, computing power, sensors, and software frameworks. The design choices are justified based on factors such as functionality, cost-effectiveness, and ease of use. Following the development process, the thesis delves into the functionalities implemented in the smart mirror prototype. These functionalities may encompass, but are not limited to: • Information Display: Real-time access to weather forecasts and news headlines.
• Interactive Features: Sentiment recognition, hand gesture recognition for interaction. The thesis then evaluates the effectiveness of the developed prototype through functionality testing. The results of these tests provided valuable insights into the effectiveness of the designed system and identified any potential hardware limitations. This information is 0crucial for future development and refinement of the smart mirror prototype.
The final section of the thesis discusses the findings and contributions of the research. It analyzes the success of the implemented functionalities, testing results of the prototype, and potential future applications of smart mirror technology. It concludes by highlighting the potential impact of smart mirrors on our daily lives, emphasizing their ability to streamline routines, improve productivity, and enhance user experience in the modern home environment. iv TABLE OF CONTENTS NHIỆM VỤ ĐỒ ÁN TỐT NGHIỆP.
iv TABLE OF CONTENTS .v LIST OF TABLES. vii LIST OF FIGURES. viii LIST OF ACRONYMS. xi CHAPTER 1 OVERVIEW .2 Scientific and practical significances .5 Structure of the report.
5 CHAPTER 2 LITERATURE REVIEW .1 Smart Mirror Technologies .5 Machine Learning for Sentiment Analysis, Hand Gesture Analysis and Song recommendation .2 Hand Gesture Analysis .6 Introduction to automatic speech recognition .7 Introduction to feature extraction technique use for speech recognition model .2 Nyquist-Shannon sampling theorem .8 Introduction to deep learning model used in automatic speech recognition .1 Convolutional Neural Networks (CNN) .3 Connectionist Temporal Classification (CTC) .9 Introduction to metrics used in automatic speech recognition .1 Word Error Rate (WER).2 Character Error Rate (CER). 29 CHAPTER 3 DESIGN HARDWARE AND ELECTRONICS SYSTEM .1 Hardware design consideration .2 Electronics system consideration .1 Design the mirror frame .2 Design the electronics system. 36 CHAPTER 4 HUMAN COMPUTER INTERACTION .1 Proactive Personalization through Sentiment Analysis .2 Intuitive Audio Control through Hand Gestures .3 Voice Command Integration: Expanding Functionality .1 Analysis of user’s visual signal .2 Analysis of user’s auditory signal – Speech to text algortihm. 61 CHAPTER 5 GENERATING MIRROR’S RESPONSE .1 Generating mirror’s response based on user’s hand gestures .2 Generating mirror’s response based on user’s emotion.1 Song recommendation pipeline .2 Distribute songs into clusters.3 Serve recommended songs to users .3 Generating mirror’s response based on user’s speech.
78 CHAPTER 6 EXPERIMENTAL RESULTS. 79 CHAPTER 7 CONCLUSION AND DISCUSSION. 83 vi LIST OF TABLES Table 2.1: Comparison of hand gestures recognition methods.1: Power consumption of components .2: Comparison of IR proximity sensors .3: Comparison of different kinds of cameras .4: Comparison of RealSense of cameras .1: Whisper models comparison .2: Quantization and PEFT result. 70 vii LIST OF FIGURES Figure 1.2: System’s response of “Smart mirror for smart life” project (Source: [3]) .1: Reflection and Image Formation by mirrors (Source: [5]) .2: Mirror with display and touchscreen capabilities (Source: [6]) .3: Smart mirror with facial recognition technology (Source: [7]) .5: Time-of-Flight (ToF) camera’s working principle (Source: [10]) .6: Depth camera based light pattern distortion analysis .7: Integration of Embedded System in Smart Mirror (Source: [11]) .8: Fundamental states of Sentiment Analysis (Source: [13]) .9: Ahmet Gunduz-Neslihan Kose’s approach for hand gestures recognition (using a sequence of images as input and a CNN classifier) .10: Alam - Islam - Rahman’s approach for hand gestures recognition (using a single image as input and a CNN - VGG16 based classifier model) .11: Using k-means algorithm to cluster data points (Source: [16]) .12: An example of spectrogram (Source: [17]) .13: Basic CNN architecture (Source: [19]) .15: The CTC architecture (Source: [21]) .1: The proposed main frame of the mirror .2: The proposed placement of components inside the frame (front view and back view) .3: The completed main frame of the mirror .4: IR sensor placed beside the frame .5: The proposed camera bracket .6: Camera bracket placed on top of the frame .7: Block diagram of Electronics system .8: Electrical system of the project .9: AC 220V to DC Converter Driver 12V-10A .10: The proposed design of the electrical box .11: The completed electrical box with all components inside .12: Jetson Nano specifications (Source: [23]) .13: Circuit design for integrating a Proximity Sensor for Keyboard activation .16: Distribution of sensors and cameras on RealSense camera .17: Hand gesture recognition module run with depth filter RealSense camera .2: Flow chart of Sentiment analysis .3: Octavio's facial expression classification architecture (Source: [28]) .4: Training/Validate accuracy and loss .5: Media's hand tracking pipeline .6: 21 key points on the hand (Source: [31]) .7: Translating hand pose to user's command pipeline .8: Block diagram of identifying raised fingers algorithm .9: Play and pause gesture .10: Skip to next audio gesture (left) and skip to previous audio gesture (right) .11: Volume adjusting gesture .12: The angle between three landmarks 5, 0 and 17 (left) and the angle of landmarks 9,0 with respect to the horizontal line (right).13: Flow chart for ASR system .14: Flow chart for audio signal input data feature extraction .15: Model quantization schemes (Source: [33]) .16: Demonstration of LoRa method (Source: [35]) .18: Demonstration for Multi-head Attention .19: Whisper Small Training/Eval Loss .1: Components of application’s graphical user interface .2: Music recommendation pipeline .3: 200 most popular songs of the week (Source: [37]) .4: Output graph of the elbow method .5: Sample output of k-means method .6: user responds based on user's speech .