MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT AUTOMATION AND CONTROL ENGINEERING TECHNOLOGY DESIGN AND IMPLEMENTATION OF ATTENDANCE AND STUDENT MONITORING SYSTEM USING IMAGE PROCESSING AND ARTIFICIAL INTELLIGENCE SUPERVISOR: LE MY HA, Assoc.Prof STUDENT: BUI MINH TRI SKL 0 0 9 1 6 7 Ho Chi Minh City, August, 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT DESIGN AND IMPLEMENTATION OF ATTENDANCE AND STUDENT MONITORING SYSTEM USING IMAGE PROCESSING AND ARTIFICIAL INTELLIGENCE BUI MINH TRI - 18151041 MAJOR: AUTOMATION AND CONTROL ENGINEERING TECHNOLOGY ADVISOR: LE MY HA, Assoc.Prof Ho Chi Minh City, August 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT DESIGN AND IMPLEMENTATION OF ATTENDANCE AND STUDENT MONITORING SYSTEM USING IMAGE PROCESSING AND ARTIFICIAL INTELLIGENCE BUI MINH TRI - 18151041 MAJOR: AUTOMATION AND CONTROL ENGINEERING TECHNOLOGY ADVISOR: LE MY HA, Assoc.Prof Ho Chi Minh City, August 2022 APPENDIX 3: (Graduation Project Assignment) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, August , 2022 GRADUATION PROJECT ASSIGNMENT Student name: Bui Minh Tri Student ID: 18151041___________________ Major: Automation and control engineering Class: 18151CLA2 technology Advisor: Assoc. Prof Le My Ha Phone number: 036.8404 Date of assignment: Date of submission: 1. Project title: Design and implementation of attendance and student monitoring system using image processing and artificial intelligence. Initial materials provided by the advisor: - Image processing and machine learning documents such as papers and books - The related thesis of previous students 3.
Content of the project: - Refer to documents, survey, read and summarize to determine the project directions - Survey to choose the suitable model - Write programs for Jetson Nano - Test and evaluate the completing system - Design the graphical user interface - Upload and retrieval of data to the database - Send email warning for absent students or students have the cumulated time is shorter than the standard time - Collect new data and retrain the classifier - Write a report - Prepare slides for presenting 4. Final product: The hardware and software of an attendance system, real-time database, user interface CHAIR OF THE PROGRAM ADVISOR (Sign with full name) (Sign with full name) i APPENDIX 4: (Advisor’s Evaluation sheet) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, August , 2022 ADVISOR’S EVALUATION SHEET Student name: Bui Minh Tri Student ID: 18151041 Major: Automation and control engineering technology Project title: Design and implementation of attendance and student monitoring system using image processing and artificial intelligence Advisor: Assoc. Prof Lê Mỹ Hà EVALUATION 1. Content of the project: - The thesis has a total of six chapters with 61 pages - The construction and design of attendance and student monitoring system, in the particular room.
- The real system is completed following the objectives in the proposal. Strengths: - The system can support the teachers and administration in terms of taking attendance and monitoring students who go into or go out of the class. - The system is designed with image processing and artificial intelligence. - This project is low-cost.
- The execution time is suitable for practical application. - The accuracy of the system is guaranteed. Weaknesses: - The system is not tested in different environments and with a limited source of dataset. Different environments require different camera positions to avoid the backlit.
Approval for oral defense? (Approved or denied) .) Ho Chi Minh City, month day , year ADVISOR (Sign with full name) ii APPENDIX 5: (Pre-Defense Evaluation sheet) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, August , 2022 PRE-DEFENSE EVALUATION SHEET Student name: Bui Minh Tri Student ID: 18151041 Major: Automation and control engineering technology Project title: Design and implementation of attendance and student monitoring system using image processing and artificial intelligence Name of Reviewer:. Content and workload of the project. Approval for oral defense? (Approved or denied) .) Ho Chi Minh City, month day , year REVIEWER (Sign with full name) iii APPENDIX 6: (Evaluation sheet of Defense Committee Member) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name:. Name of Defense Committee Member:.
Content and workload of the project .) Ho Chi Minh City, month day , year COMMITTEE MEMBER (Sign with full name) iv ACKNOWLEDGEMENT We would like to sincerely thank Professor Le My Ha for his thorough instruction, which helped us to have the necessary information to use for completing the thesis. During the whole progress, even if we did our best to complete everything completely, mistakes are still inevitable. We anticipate having my advisor's focused assistance and direction to help us gain more experience and successfully complete the topic project. On the other hand, we would like to express our sincere thanks to the Faculty of Hight Quality Training and Faculty of Electrical and Electronics Engineering where we obtained basic knowledge and experience.
Moreover, we also would like to thank ISLab members who helps us in detailing the works of this project. They shared valuable experience and knowledge with us. Ultimately, we would like to express our gratitude to our families for their support of our team throughout the implementation of this thesis. Sincere thanks for everything! v GUARANTEE This thesis is the result of our study and implementation, which we hereby formally proclaim.
We did not plagiarize from a published article without author acceptance. We will take full responsibility for any violations that may have occurred. Authors Bui Minh Tri vi ABSTRACT Face recognition is a computer application that detects, tracks, identifies and verifies human faces in images or videos captured with a digital camera. There has been significant progress in the field of face detection and recognition for security, identification, and attendance purposes, we are inspired by that to apply face recognition technology to every walk of life.
In the university, we witness teachers take a lot of time to take students’ attendance. Moreover, some attendance systems in the workspace are not suitable for taking attendance of students. All reasons mentioned above are the motivation for me to do this project. Furthermore, we also propose this solution to substitute the traditional attendance system because of our proactive recognition.
The attendance system is embedded in Jetson Nano because we want to design an affordable system. We utilize face detection and recognition functions that are available on traditional systems and add the face tracker. To manage information, we update all recorded data into the server database. For users to operate easily, we design a graphical user interface.
The experiment is produced in a particular room simulates a practical lesson, with limited identities in the dataset. The lowest frame per second of the system is 8fps but still satisfies real-time practical applications. vii CONTENTS ACKNOWLEDGEMENT. vii LIST OF FIGURES.
xii LIST OF TABLES. xv CHAPTER 1: INTRODUCTION. Aims and objectives. 2 CHAPTER 2: LITERATURE REVIEW.
Convolutional neural network. Fully connected layer. Multi-scale feature maps. Forecast bounding box.
Non-max suppression. Cross-stage partial connection (CSP). Spatial pyramid pooling (SPP). Path aggregation network (PAN).
Spatial attention model (SAM). Cross mini batch normalization (CmBN). Types of object tracking. Evaluating deep learning models.
23 CHAPTER 3: HARDWARE PLATFORM. 7inch HDMI LCD. 28 CHAPTER 4: SOFTWARE DESIGN. Face landmark estimation.
Simple online real-time object tracking. Face and facemask detection. CONCLUSION AND FUTURE WORK. 59 xi LIST OF FIGURES Figure 1.
An attendance machine using magnetic cards, and biometric fingerprints. A convolutional neural network. Convolution in the convolutional layer. An illustration of convolution with stride equals 1 in CNN.
An example using zero padding. The graph of ReLu function. An example of using ReLu. An example of using max pooling.
The fully connected layer in CNN. The output tensor of YOLO. Multi-scale feature maps for detection. The output is 3 feature maps with different sizes 13x13, 26x26 and 52x52 respectively.
Anchor box in object detection. Formula to estimate bounding box from anchor box. The outer dashed rectangle is the anchor box of size (pw , ph ). The application of Non-Max Suppression algorithm.
The overall architecture of YOLOv4 [2]. The component blocks in YOLOv4 architecture. Cross-stage partial connection. The comparison of DenseNet and CSPDenseNet [4].
The results of Mosaic. The results of Mixup, Cutout, CutMix. Spatial pyramid pooling. On the right-hand side, SPP is in previous research.
On the right-hand side, modified SPP is in YOLOv4. PANet overall architecture. (a) Feature extractor uses the FPN architecture. (b) The new augmented bottom-up pathway is added to the FPN architecture.
(c) The adaptative feature pooling layer. (d) Two branches predict the bounding box coordinated and the object class. (e) One branch predicts the binary mask of the object. The dashed lines correspond to links between low-level and high-level patterns.
The red one is in the FPN consisting of more than 100 layers. The green one is a shortcut in the PANet consists of less than 10 layers. Spatial attention model. Spatial attention model.
(a) SAM is in previous research. (b) SAM is modified in YOLOv4. Cross mini-Batch Normalization. The description of Meanshift.
A typical confusion matrix. A typical ROC curves. Overall system connection. The actual system.
Connecting port on Jetson Nano board. The specifications of 7inch HDMI LCD. 7inch HDMI LCD. Additional computer accessories.
The comparison between different YOLO version. Leaky ReLu activation function. These feature points, located on every face, are drawn along the eyebrows, the eyes, the mouse, and the chin. One of the parts of the facial recognition process.
Training model using triplet loss. Kalman filter algorithm. The home page in GUI. This page requires the user to log in before entering other pages.
The data page in GUI. This page allows for retrieval into the database server. The displaying page in GUI. The user can monitor the whole system on this page.
The creating data page in GUI. Facial data of new members can be added to this page. Login function on the website. Main page on the website.
The structure of student list in the database. The structure of check table in the database. The structure of attendance table in the database. The list of users in the database.
CPU is divided into multiple threadings. The flowchart in thead 2. This is considered the main thread of the system. The flowchart in thead 3.
This is considered the data thread of the system 43 Figure 4. Convert the original model to Tensorrt. Face mask dataset. Labeled Faces in the Wild dataset.
Training graph of FaceNet. Training IoU and loss graph of the detection model. Evaluation graph of FaceNet on LWF dataset. Confusion matrix at K=24.
There are 10 people in the dataset. The final result. GUI requires to login before controlling and monitoring the system. Retrieval of database.
The result after updating data into Firebase. (a) The result in the attendance table. (b) The result in the check table. The result in the student list after updating.
Send the warning email. The retrieved data from the database on 17 August. 56 xiv LIST OF TABLES Table 2. The list of improvements in YOLOv4.
The specifications of Jetson Nano. Multiple kinds of affine transformation. GPU configuration in Google Colab. Training parameter of the detection model.
Training parameter of the recognition model. Evaluate average precision following each class. Evaluate TP, TN, FP, FN for confident threshold = 0.25 with average IoU = 65.