MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION GRADUATION THESIS MAJOR: COMPUTER ENGINEERING TECHNOLOGY FACIAL RECOGNITION AND HAND GESTURES TO CONTROL HOME APPLIANCES INSTRUCTOR: PHD. DO DUY TAN STUDENT: LE THI THANH THU NGUYEN TRONG HAI Ho Chi Minh city, June 2024 Faculty of International Education – HCMC University of Technology and Education HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY OF INTERNATIONAL EDUCATION GRADUATION PROJECT FACIAL RECOGNITION AND HAND GESTURES TO CONTROL HOME APPLIANCES LÊ THỊ THANH THƯ Student ID: 18119044 NGUYỄN TRỌNG HẢI Student ID: 19119028 Major: COMPUTER ENGINEERING Advisor: ĐỖ DUY TÂN, PhD. Ho Chi Minh City, June 2024 Faculty of International Education – HCMC University of Technology and Education HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY OF INTERNATIONAL EDUCATION GRADUATION PROJECT FACIAL RECOGNITION AND HAND GESTURES TO CONTROL HOME APPLIANCES LÊ THỊ THANH THƯ Student ID: 18119044 NGUYỄN TRỌNG HẢI Student ID: 19119028 Major: COMPUTER ENGINEERING Advisor: ĐỖ DUY TÂN, PhD. Ho Chi Minh City, June 2024 Faculty of International Education – HCMC University of Technology and Education Faculty of International Education – HCMC University of Technology and Education Faculty of International Education – HCMC University of Technology and Education Faculty of International Education – HCMC University of Technology and Education THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness Ho Chi Minh City, June 20, 2024 EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name: Lê Thị Thanh Thư Student ID: 18119044 Student name: Nguyễn Trọng Hải Student ID: 19119028 Major: Computer Engineering Technology Project title: Facial Recognition And Hand Gestures To Control Home Appliances Advisor: PhD.
Đỗ Duy Tân EVALUATION 1. Content and workload of the project. ) Ho Chi Minh City, month day, year COMMITTEE MEMBER (Sign with full name) Faculty of International Education – HCMC University of Technology and Education DISCLARATION This project has been thoroughly researched and implemented with diligence. No content has been directly copied from existing projects.
Proper citations have been provided for all references used. We take full responsibility for any potential violations that may arise. Authors LE THI THANH THU NGUYEN TRONG HAI i Faculty of International Education – HCMC University of Technology and Education ACKNOWLEDGEMENTS We extend our deepest gratitude to PhD. Do Duy Tan for his invaluable contributions to our project.
Tan's unwavering dedication, insightful guidance, and boundless enthusiasm have played a pivotal role in shaping the trajectory of our work. His profound expertise and generosity in sharing knowledge have not only enriched our project but also inspired us to explore new realms of creativity and innovation. Throughout the implementation process, despite our best efforts, we acknowledge that challenges and occasional mistakes are inevitable. Tan's steadfast support and mentorship have empowered us to overcome obstacles with resilience and determination, transforming setbacks into valuable learning opportunities.
His unwavering belief in our capabilities has motivated us to push the boundaries of our creativity and strive for excellence in all facets of our work. Lastly, we wish to express our heartfelt appreciation to all who have supported our group's endeavors, whether through words of encouragement, practical assistance, or simply by being a listening ear. Your steadfast support has been a constant source of motivation and inspiration, fueling our determination to succeed. Together, we have embarked on a journey of innovation and collaboration, and it is through the collective efforts of individuals like Mr.
Tan and our broader support network that we have been able to achieve our objectives and make meaningful progress towards our shared vision. Thank you all for your invaluable contributions, dedication, and camaraderie. Your support has been deeply valued, and we eagerly anticipate continuing our collaborative efforts as we work towards our common goals. ii Faculty of International Education – HCMC University of Technology and Education TABLE OF CONTENTS DISCLARATION.
ii TABLE OF CONTENTS. iii LIST OF FIGURES. vi LIST OF TABLES. viii LIST OF ABBREVIATIONS .7 THE NOVELTY OF TOPIC .1 INTERNET OF THINGS.
18 iii Faculty of International Education – HCMC University of Technology and Education 2. DESIGN AND IMPLEMENTATION .2 Raspberry Pi transmits data to Database .3 ESP32 receives data to Database .2 Image Processing Block .1 Building Custom Dataset .2 Set Up train YOLO model .2 Hand Gestures Recognition .1 Configure Parameter of Hand Gestures .2 Set Up MediaPipe .2 Control Home Appliances via Camera .3 Remote Home Appliances via GUI App .1 Performance Assessment of IoT System .2 Performance of the AI model. CONCLUSIONS AND FUTURE WORKS. 73 iv Faculty of International Education – HCMC University of Technology and Education REFERENCES.
75 v Faculty of International Education – HCMC University of Technology and Education LIST OF FIGURES Figure 2. Architecture of CNN network [1 - Figure 5]. Development time of YOLO network [2 – Figure 1]. The structure of YOLO network [3 – Figure 1].
Detailed illustration of YOLOv8 model architecture. The Backbone, Neck, and Head are the three parts of our model, and C2f, ConvModule, DarknetBottleneck, and SPPF are modules [26 – Figure 4]. Landmarks positions detected by MediaPipe [4 – Figure 5]. The mobile app development process [5].
Block diagram of system. The sequence diagram of system. The flowchart of building database. The structure of database.
API of database. The block diagram of Raspberry Pi transmit data to Firebase. The flowchart of deploy API on Raspberry Pi. The flowchart depicting the process of data uploading to the database after facial recognition.
The flowchart depicting the process of data uploading to the database after hand gestures recognition. The block diagram of ESP32 receives data to Firebase. The flowchart of deploy API on ESP32. The flowchart for the process of receiving data from the database to control door 36 Figure 3.
The flowchart for the process of receiving data from the database to control fan. The flowchart for the process of receiving data from the database to control led. The flowchart of image processing block. The flowchart of facial recognition process on Rasp.
The flowchart of hand gestures recognition process on Rasp. Labelling each face of custom dataset. The custom dataset after labelling. Choose type of dataset to export dataset.
The custom dataset after export from Robotflow. The sample image of custom dataset with bounding box. Information about example bounding box. The flowchart of YOLO model installation on Raspberry Pi.
48 vi Faculty of International Education – HCMC University of Technology and Education Figure 3. The resulted images after test model. The flowchart of deploy YOLOv8 model on Rasp. The flowchart of set up MediaPipe on Rasp.
The flowchart of control block. The flowchart of GUI. Result of external system design. Results of internal system design.
Result of facial recognition on Rasp then transmit data to database. Result of control solenoid lock after ESP32 receives data from database. Result of index finger recognition on Rasp then transmit data to database. Result of thumb finger recognition on Rasp then transmit data to database.
Result of control fan after ESP32 receives data from database. Result of pinky finger recognition on Rasp then transmit data to database. Result of ring finger recognition on Rasp then transmit data to database. Result of control light after ESP32 receives data from database.
Results of remote door via GUI app. Results of remote Fan via GUI app. Results of remote Led via GUI app. Results of controlling home appliances via the GUI app.
67 vii Faculty of International Education – HCMC University of Technology and Education LIST OF TABLES Table 2. Camera comparison table. Mini computers comparison table. Hand Gestures for User Manual.
The state table of hand gestures recognition. Evaluate average processing time in daylight condition. Evaluate average processing time in nightlight condition. Comparison table of between YOLOv5 and YOLOv8.
Comparison table of between MediaPipe and Handtrack. 70 viii Faculty of International Education – HCMC University of Technology and Education LIST OF ABBREVIATIONS AI Artificial intelligence YOLO You Only Look Once CV Computer Vision CNN Convolutional Neural Network IoT Internet of Things GUI Graphical User Interface WI-FI Wireless Fidelity API Application Programming Interface URL Uniform Resource Locator ix Faculty of International Education – HCMC University of Technology and Education ABSTRACT This study presents an innovative system that combines advanced facial recognition technology for door access with precise hand gesture recognition for controlling lighting and ventilation systems. The system's design aims to address the dual objectives of enhancing security measures and streamlining user interactions within controlled environments. Facial recognition technology serves as the primary means of access control, allowing authorized individuals to gain entry to restricted areas by simply presenting their face to the system's camera.
This streamlined authentication process eliminates the need for physical keys or access cards, enhancing convenience and efficiency for users. In parallel, hand gesture recognition technology provides an intuitive interface for controlling lighting and ventilation systems. By recognizing specific hand gestures, users can seamlessly adjust lighting levels and fan speeds without the need for physical switches or controls. This intuitive interaction mechanism enhances user experience and promotes energy efficiency by facilitating precise control over environmental conditions.
The integration of facial recognition and hand gesture recognition technologies offers a comprehensive approach to access control and environmental management in diverse settings. Beyond traditional security applications, the system's versatility makes it suitable for various environments, including residential, commercial, and industrial spaces. Implementation of this integrated system is anticipated to yield significant benefits, including improved security, enhanced user convenience, and optimized energy utilization. Furthermore, the insights provided in this report serve as valuable resources for students and professionals in Computer Engineering, Telecommunication Electronic Engineering, and related fields, offering detailed perspectives on system architecture, implementation strategies, and potential applications in real-world scenarios.
x Faculty of International Education – HCMC University of Technology and Education Chapter 1. INTRODUCTION The rapid evolution of IoT technology has profoundly reshaped daily life, with home automation emerging as a pivotal advancement. These systems empower users to effortlessly manage household appliances, bolstering comfort, security, and energy efficiency. Integrating facial recognition and hand gestures into IoT home automation offers a novel way to personalize access and streamline control methods.
This research explores developing an IoT-based home appliance control system using these technologies to enhance security, efficiency, and convenience in everyday living.1 INTRODUCTION The incorporation of face and hand gesture detection AI technologies into IoT-based smart home systems adds a number of novel features and benefits that set this project apart from other smart home solutions. This idea takes use of many major areas of originality. First, it uses a dual recognition system: face recognition for individualized access and seamless verification, as well as hand gesture detection for contactless, intuitive operation. This combination provides a frictionless, sanitary, and easily accessible interface that improves user convenience and security.
The improved user interface offers smooth interaction and an adaptable environment, making device operation intuitive and adapted to individual preferences. Users may manage their home equipment using natural gestures and facial expressions, such as finger up to turn on lights or fan, removing the need for physical connection with the devices. Furthermore, this initiative focuses on enhanced interoperability, connecting with current IoT devices to form a coherent, networked home environment that may adapt in response to future technology advances. The system's ability to integrate with a variety of smart home devices guarantees a comprehensive and integrated experience.
This interoperability enables a uniform control interface, which simplifies user interactions across many devices and platforms, hence improving the overall user experience. Improved security and privacy are also important, with face recognition offering safe access control and on-device processing protecting data privacy.