VIETNAM NATIONAL UNIVERSITY, HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY FACULTY OF COMPUTER ENGINEERING PHAM HO NGOC BINH TRAN HUU NHI GRADUATE THESIS RESEARCH AND IMPLEMENTATION OF SAFETY WORKER-DETECTING CAMERA FOR ENGINEER OF COMPUTER ENGINEERING HO CHI MINH CITY, 2021 VIETNAM NATIONAL UNIVERSITY, HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY FACULTY OF COMPUTER ENGINEERING PHAM HO NGOC BINH - 16520097 TRAN HUU NHI - 16521761 GRADUATE THESIS RESEARCH AND IMPLEMENTATION OF SAFETY WORKER-DETECTING CAMERA FOR CONSTRUCTION SITE NGHIEN CUU VA HIEN THUC CAMERA NHAN DANG CONG NHAN DAM BAO AN TOAN LAO DONG TRONG CONG TRUONG XAY DUNG ENGINEER OF COMPUTER ENGINEERING MENTOR Ph. NGUYEN MINH SON HO CHI MINH CITY, 2021 LIST BOARD OF PROTECTED THESIS Council of graduation thesis reviews established under Decision No 70/QD- DHCNTT, date January 21 2020 by Principal of University of Information Technology. ACKNOWLEDGEMENTS First, my group would like to thank to the teachers of Faculty of Computer Engineering, University of Information Technology, Ho Chi Minh city for giving us opportunities to do this thesis. Specially, we would like to give our sincere and gratitude to our mentor, PhD Nguyen Minh Son.
Thank you for guiding and supporting us during the time of making thesis. Thank to our friends and students of the previous courses who give us more ideas and knowledge to do this thesis. We have learned lots of valuable experiences from reality and technology during the process of making thesis. However, we can’t avoid making mistake so we hope the council will forgive us.
Thank you sincerely! Representative student PHAM HO NGOC BINH - TRAN HUU NHI Faculty of Computer Engineering - 2021 TABLE OF CONTENT Chapter 1. LG HH TH HT TH TH HH HH 3 1. Construction safety in Viet ÏNa1m. --GG Q TH TH H H HH ntt 4 1.
Goal Of th€SIS.Qn Hnn TH g nHH T HHH nkrh 5 Chapter 2. Object ef€CtIOI. LH HH HH HH kkt 6 2. HH TH TH HH 6 2.
LH HT HH HT HH HH HH 8 2. Why choosing YOLO:. Object detection’s Support fOOÏS:. HH HH ng ng 15 2.
Darknet Íraim€WOTK:.-- Ăn TH HH ng 16 2. Hardware 1mpleImen(atIOII. 5 5 +13 1kg ng ệt 17 2. Nvidia Jetson Nano Developer KT(.-- cv HH ng ng ệt 20 "` ®.
PPE SYSTEM ANALYSIS AND DESIGN. Designing PPE detection and monitoring system:. Creating PPE mOel:. -- <5 + xxx vn HH nh HH tư 24 3.
Selecting PPE datfaset:. Creating PPE training dafase€t:. Training PPE’s model:. Building and deploying real-time detection and monitoring PPE system:.
Install Jetson nano OS and necessary lIbrarles:. Install Yolov4 from Darknet framewWOrK:. Implement and run detecting and monitoring system:. Packaging camera box for detection system? .- LG LH ST tk ky 44 4.
Evaluate trained PPE model. Testing in real CAS©:.c- 2c 2c HH HH HH He 45 4. Full of light condition accuracy r€SuÌ(S:. Lack of light condition accuracy resuÏ(S:.
CONCLUSION AND FUTURE WORK. 2G TH HH gi 53 bố ha. Future Work: oo eee cscssecssecseessecesecseesseesseessecseessecssecssecseeseesaeessesseeeaeens 54 3581545105070. 55 TABLE OF FIGURE Figure 1.1: Inspection in COMStruction Site .2: Workers in COMStrUCTION Site .1: Objects detected using a YOLOv3 model trained.- -- G11 1911911991011 HH HH ng 7 Figure 2.3: YOLOv4-tiny network structure [Ố].4: Example of how Yolo WOTKS.5: Average precision and speed on different model.
1H TH TH TH HH ng 13 I200i02/40)0409)0.8: Darknet framework 1COH. c6 + 1311831831118 311 1 91 1991 1v vn ng ng 16 Figure 2. c1 9111110113111 9101 91 HH nh 17 Figure 2.11: Jetson Nano Developer KT(.- ------s + sc + +svseeerseeeeeseeereere 18 Figure 2.12: Performance of various deep learning inference networks [15].13: Raspberry Pi Camera V2.14: Remote laptop by smart phone using VNC.1: PPE detection and monitoring system diaøTam.-- -- - + + +11 *31 18311189111 91111 11 1 vn ng 24 Figure 3.3: Hard hat for COMStruction SIfC.4: Safety goggles for CONStrUCTION SI{€.5: Yellow visibility safety V€SẨ.- sàn HH nh HH HH ưệt 26 Figure 3.6: Orange visibility safety V€SÍ. HH HH HH net 27 Figure 3.- Ăn TH HH TH TH HH kh 28 Figure 3.8: Images of worker wearing safety vest and hard hat while working.9: Image of worker wearing fully PPE.-- --- «+ sx£eseseeseeseeseeske 29 Figure 3.10: Hard hat images.11: Our self-captured image in indoor rOO.-- 5s «+ +s£+sx+s£+sx2 30 Figure 3.12: Marking bouding boxes of object using Labellmg.13: Structure of data for traInINBE,.--- «nh rưệt 32 Figure 3.14: Into “data” fOÏ€T.-- ¿- 6 +1 23 23191911 9191 vn ng HH ng gưệp 32 Figure 3.15: Training on Google COlab.
-- 5 - << 311v TH ng trên 35 Figure 3.16: all yolov4-tiny models were generated after training.17: Chart of mAP and average loss after training.18: Inserting micro-SD card to Jetson nano SD slot.19: Plug in Camera Raspberry Pi V2 to Jetson Nano.20: Real-time detecting and monitoring system điagram.21: Flow chart of detecting and monitoring sysfem.22: Results of SÿSf€I.-- 5 «S1 TH TT ng41 Figure 3.23: Camera box’s elements.24: Adapter Philips SV 4A .- ác HS HH HH HH HH ng rệt 42 Figure 3.- tk TT TH ngàng Hưng hư 43 Figure 4.1: Evaluate of mode using testing S€( .2: Test scenario of detecting ability at different distances.3: Chart of accuracy rate of PPE’s object in full of light condition.4: Accuracy rate at 1m5 (full of li h†).----s «+ +< + s++sscx+seexseesssers 47 Figure 4.5: Accuracy rate at 2m5 (full of lipht).-- 55+ +-<ss++sex+seexseeesseers 47 Figure 4.6: Accuracy rate at 3.-- -¿- «+ ss++s£+s£+eeeseesessesses 48 Figure 4.---- «xe seEeeeeeeeserserserske 48 Figure 4.8: Accurate rate at “7IH. G1 111011911 ng HH nh 49 Figure 4.9: Accuracy rate at 251m.10: : Chart of accuracy rate of PPE’s object in lack of light condition.11: Accuracy rate at distance 1.12: Accuracy rate at distance 2.13: Speed of detection when 6 objects are detected. --- -----«- 52 TABLE OF TABLES Table 2.1: Jetson nano specification .1: Information of collected training images Table 3.2: Number of images for PPE detection training and testing. LIST OF ACRONYMS CNN: Convolutional Neural Networks COCO dataset: Common Objects in Context FPS: frame per seconds GFLOPS: Giga Floating Point Operations Per Second GPU: Graphics Processing Unit IOU: Intersection Over Union GUI: Graphical User Interface GPIO: general-purpose input/output mAP: mean average precision PPE: personal protective equipment.
SSD: Single Shot Detector. VNC: Virtual network computing YOLO: You Only Look Once ABSTRACT Object detection is computer vision and image processing technology base on ability of computer systems and software to locate objects and recognize each object in an image or video. Object detection has been widely used in computer vision tasks such as face detection, vehicle detection, pedestrian counting, driver assistant, security systems,. In our thesis, we used object detection to check workers safety by monitoring personal protective equipment (PPE) of workers in construction site environment.
The real-time video detection system is still a challenging. First, the construction site environment consists multiple complications such as different illumination levels, a wide range of personal protective equipment colors and designs. We collected huge data of PPE images for training to reduce loss value. Second, object detection tasks need strong resource GPUs to parallel computing and make system fast.
Thank to Nvidia for launching Jetson Nano Developer Kit, things are getting easier. It is a tiny, powerful computer, but a fair price is still guaranteed. Therefore, we have built project and deploy it into Jetson Nano Developer Kit. It is suitable for real-time detection and advancing future research in automation in construction.
Third, we use YOLO algorithm for real-time object detection. It has very good results on the COCO dataset with provides large-scale object detection, segmentation, caption dataset and results or relationship between inference time and accuracy of deep learning algorithms. PREFACE Construction site is one of the most dangerous places that records high number of annual accidents. There are many solutions to ensure safety and minimize accidents for workers, one of which is to ensure workers wear appropriate personal protective equipment (PPE) specified in safety regulations.
However, the monitoring of PPE is costly, ineffective and time-consuming because it is still mainly based on manual inspection. Therefore, we have built a computer system which can detect personal protective equipment (PPE) in images or videos captured through camera. The obtained detection accuracy for 4 PPE’s object up to 90%. The results have demonstrated the ability to detect PPE with high precision in real-time and shown on screen to help the contractors monitor and ensure worker’s safety.
It gives much more solutions with accurate and safety management system than mainly management. It also can process 10 frames per second (FPS) on embedded computer (Jetson nano) which suitable for real-time. Construction safety in Viet Nam In Vietnam, according to the Report of Ministry of labor-invalids and social affairs in 2019, there were 8150 occupational accidents nationwide, causing 8327 victims, of which most of case involved not wearing personal protective equipment. To ensure constriction worker’s safety a lot of several onsite safety regulations have been established.
In those regulations, the appropriate use of appropriate PPE is specified and the contractors must ensure that the regulations are enforced through the monitoring process. The monitoring of PPE is normally conducted in site entry and one site construction field. It is manually monitored by inspectors. Therefore, it becomes more time-consuming and costly because a huge workers of construction Figure 1.1: Inspection in construction site 1.
Domestic research Domestically, there are a lot of research project about object detection. Some of them were applicated in license plate identification, vehicle detection, but none of them were used in monitoring of PPE in the construction site. The construction sites still conduct the monitoring of PPE using manually by contractors or inspectors with infrequent frequency. This work is ineffective and time-consuming due to the high Figure 1.2: Workers in construction site 1.
Foreign research There are some technologies have been released to enhance the construction safety. Computer vision has been widely used in [2], [3] (See more in References [2], [3]). However, most of them only focus on detecting the use of hardhat on worker. Besides hardhat, others equipment such as safety vest, safety goggles and gloves are equally important.
Safety vest has high visibility and reflectivity which can help worker avoid colliding with the individual. Safety goggles and gloves protect worker’s eyes and hands. Goal of thesis The objective of this thesis is to build a safety worker-detecting camera system, using embedded computer which support strong GPUs to combine with YOLO (CNN algorithms) to bring high speed and high accuracy. The system will detect and monitor PPE [hardhat, safety goggles, safety vest, gloves] of workers when they come in construction site.
We also create dataset of PPE object for future working. The system will have: - Detecting 4 kind of PPE including: hard hat, goggle, safety vest, gloves. - Monitoring safety of worker by send notification on screen PASS and NOT PASS status and saving results of detection (object detection, accuracy rate of objects) and image of detection. Limitation of thesis: - Hardware: Jetson Nano, Camera Raspberry Pi v2.
- Algorithm: YOLOv4-tiny. - PPE’s object: HARD HAT, GOGGLES, SAFETY VEST, GLOVES. - Speed: We try to optimize the system to be run at least (8 — 10) FPS. - Accuracy rate: 80% — 90% of accuracy rate.
Overview: Object detection is a field of Computer Vision and Image Processing. Object detection refers to the ability of computer system and software to locate objects and identify each object of a certain class (like humans, buildings, vehicles, book, chair, etc.) in digital images and video. Well-researched domains of object detection include Face detection, Object tracking, Activity recognition, Pedestrian counting, Security Figure 2.1: Objects detected using a YOLOv3 model trained.