NGUYỄN TRUNG KIÊN DEVELOPING A REAL-TIME OBJECT DETECTION SYSTEM ON FPGA LUẬN VĂN THẠC SĨ NGÀNH ĐIỆN TỬ, NĂNG LƯỢNG ĐIỆN, TỰ ĐỘNG HÓA CHUYÊN NGÀNH KỸ THUẬT TRUYỀN THÔNG VÀ DỮ LIỆU NGƯỜI HƯỚNG DẪN KHOA HỌC TS. BÙI DUY HIỂU TS. TRẦN THỊ THÚY QUỲNH PGS TS. ERWAN LIBESSART HÀ NỘI, NĂM 2024 DEVELOPING A REAL-TIME OBJECT DETECTION SYSTEM ON FPGA Master thesis of Paris-Saclay University and VNU University of Engineering and Technology Specialization: M2 Data and Communication Engineering Research unit: Advanced Institute of Engineering and Technology, VNU University of Engineering and Technology Thesis presented at Hanoi, on 22 January 2024 NGUYEN Trung Kien Committee Arnaud BOURNEL Paris-Saclay University Chairman Pierre DUHAMEL CNRS, CentraleSupelec, Paris-Saclay University Rapporteur NGUYEN Linh Trung VNU University of Engineering and Technology Examiner BUI Duy Hieu VNU Information Technology Institute Examiner Erwan LIBESSART CentraleSupelec, Paris-Saclay University Examiner Thesis Supervision Master Thesis BUI Duy Hieu VNU Information Technology Institute Supervisor.
TRAN Thi Thuy Quynh VNU University of Engineering and Technology Co-supervisor. Erwan LIBESSART CentraleSupelec, Paris-Saclay University Co-supervisor. ABSTRACT Real-time object detection is now a key capability for embedded computer vision platforms used in numerous applications, from surveillance to transportation and defense systems. However, the limited resource constraints and demands for low- latency accurate detection impose significant optimization challenges.
The key motivation is enabling real-time object detection using lightweight algorithms on embedded devices without powerful specialized hardware. The pipeline combines selective background subtraction using Zipfian Estimation techniques with an optimized HOG-SVM classifier to minimize computations. Additional techniques like multi-scale scanning via parallel classification threads are employed to enhance throughput. This thesis proposes a real-time object detection system for embedded devices with stationary cameras that uses lightweight algorithms.
It includes an overview of related work, presents the proposed system, and aims to evaluate its effectiveness through simulation tests on software written in C++ and videos from the MOT15 dataset. The system achieved frame rates between 15-24 FPS, demonstrating efficiency. However, the accuracy metric, mAP, showed that simpler non-neural methods resulted in less than 10% accuracy, highlighting the tradeoffs. While further development is required, this thesis provides a promising foundation for real-time detection that balances efficiency and accuracy and aims to be implemented into hardware.
Keywords: Object detection, Real-time, Background subtraction, Zipfian estimation, HOG features, SVM classification, Parallel processing, Multi-scale. AUTHORSHIP I have completed my graduation thesis, “Developing a Real-Time Object Detection System on FPGA” as a result of my labor under the guidance of my supervisors. Within the reference section of my thesis, I have correctly identified all sources used in its creation. I can attest that all data and results this thesis presents are accurate.
I alone am responsible for any errors that may be found, subject to any disciplinary measures imposed by the University Paris-Scalay and the University of Engineering and Technology, Vietnam National University, Hanoi. Hanoi, January 22nd, 2024 Author ACKNOWLEDGEMENT I want to express my sincere gratitude to the professors, students, staff, and institutions that played an essential role in my success throughout my Paris Saclay M2 Master's program in Communication and Data Engineering that cooperated with the VNU University of Engineering and Technology and Université Paris-Saclay. I especially want to thank my supervisors, Dr. BUI Duy Hieu from VNU Information Technology Institute, Dr.
TRAN Thi Thuy Quynh from VNU University of Engineering and Technology, and Dr. Erwan LIBESSART from CentraleSupélec for supervising my master's thesis. TRAN Xuan Tu – Director of VNU Information Technology Institute, for providing valuable feedback. Completing this thesis was only possible with their guidance and support.
I also want to thank esteemed professors: Prof. Pierre DUHAMEL at Centrale Supélec, Prof. NGUYEN Linh Trung and Dr. NGUYEN Hong Thinh at VNU University of Engineering and Technology, and Prof.
Arnauld BOURNEL at Université Paris-Saclay. Their guidance was crucial for my success during my Paris Saclay M2 Master's program, and I am confident that it will help me greatly in my future career. Furthermore, I would like to thank my manager, MSc. DANG Van Huan at Viettel Aerospace Institute, thank you for providing me with the necessary support to complete my master's program.
Lastly, I would like to express my heartfelt appreciation to my family, girlfriend, friends, and colleagues for their unwavering support throughout this journey. I sincerely appreciate your help and am always grateful for your support. iii List of Abbreviations. vi List of Figures.
viii List of Tables. Real-time Object Detection System. Object Detection Overview. Real-time Object Detection Challenges.
Proposed Real-time Object Detection System. Proposed System using Zifian Estimation Technique and HOG SVM Algorithm. Lightweight Motion Detection Algorithm used on a stationary camera. Object Detection using the HOG SVM Algorithm.
State of the Art. Object Detection Block. Implementation and Evaluations. Experiment setup environment.
Mean Average Precision. Input: TUD-Stadtmitte [40]. Input: TUD-Campus [41] .33 Conclusions and Perspective .37 List of Abbreviations Abbreviation Definition 2D Two Dimensional ADAS Advanced Driver Assistance Systems AI Artificial Intelligence AMD Advanced Micro Devices mAP Mean Average Precision ASIC Application Specific Integrated Circuit AVC Advanced Video Coding CBAM Convolutional Block Attention Module CNN Convolutional Neural Network COCO Common Objects in Context CPU Central Processing Unit DMA Direct Memory Access FPGA Field Programmable Gate Array FPS Frames Per Second GB Gigabyte GE Gate equivalents GPU Graphics Processing Unit HD High Definition HOG Histogram of Oriented Gradients LTS Long Term Support MJPEG Motion JPEG MOT15 Multiple Object Tracking Benchmark 2015 MOTC Multiple Object Tracking Challenge MPSoC MultiProcessor System On Chip NMS Non-Maximum Suppression OS Operating System PASCAL Pattern Analysis Statistical Modelling and Computational Learning PETS09 Performance Evaluation of Tracking and Surveillance 2009 PTZ Pan-Tilt-Zoom RAM Random Access Memory RCNN Regional Convolutional Neural Network RGB Red Green Blue SIFT Scale-Invariant Feature Transform SPP Spatial Pyramid Pooling SRAM Static Random Access Memory SSD Single Shot Detector SVM Support Vector Machine TSMC Taiwan Semiconductor Manufacturing Company UHD Ultra High Definition VNU Vietnam National University VOC Visual Object Classes YOLO You Only Look Once ZCU106 Zynq UltraScale+ ZCU106 Evaluation Kit List of Figures Figure 1 - Surveillance and Public Safety Camera [3]. 2 Figure 2 - Autonomous surveillance along the border in the USA [4].
2 Figure 3 - Dash Camera [6]. 4 Figure 4 - Imaging Infrared Seeker on the Missile [7]. 4 Figure 5 – Deep learning method [16]. 5 Figure 6 – Hand-crafted feature method [16].
5 Figure 7 - Pre-processing data flow. 10 Figure 8 – Processing data flow. 11 Figure 9 – Post-processing data flow. 12 Figure 10 – Proposed system architecture.
12 Figure 11 - Decomposition of a vector into the form of two vectors [35]. 16 Figure 12 - Object detection system architecture [23]. 17 Figure 13 - Proposed block diagram of object detection [24]. 17 Figure 14 - Block diagram of the proposed human detection system [36].
18 Figure 15 - Bilinear Interpolation Scale Generator (a) and illustration (b). 19 Figure 16 - HOG-SVM algorithm for human detection [23]. 20 Figure 17 – Non-Maximum Suppression [37]. 23 Figure 18 – Test System 25 Figure 19 - Intersection over Union [38] 27 Figure 20 - The image extracted from input PETS209-S2L.
29 Figure 21 - Precision, recall, and mAP graphs per frame of input PETS209-S2L. 29 Figure 22 - The image extracted from the input TUD-Stadtmitte. 30 Figure 23 - Precision, recall, and mAp graphs per frame of input TUD-Stadtmitte. 30 Figure 24 - The image extracted from the input TUD-Campus.
31 Figure 25 - Precision, recall, and mAp graphs per frame of input TUD-Campus. 32 List of Tables Table 1 - Sine and Cosine quantized value. 21 Table 2 - Truth Table Confusion Matrix. 26 Introduction Real-time object detection is an essential computer vision application used in various fields.
It enables automatic recognition of people in public areas and specialized systems used in military applications. Real-time object detection is the capability of computer vision systems to locate and classify objects in images or videos quickly enough for use in practical applications. The crucial challenges for real-time object detection are accuracy, throughput, and computation complexity. Real-time applications are frequently utilized in mobile embedded systems with limited memory, computation power, and battery life.
There are two main types of object detection algorithms: the traditional method, which uses feature extraction algorithms, and deep learning, which uses neural networks. Deep learning methods can be further categorized into two types. The first type consists of region proposal object detection algorithms, which generate region proposal networks and classify these regions afterward. Examples of these algorithms include RCNN, SPP-net, Fast-RCNN, and Faster-RCNN.
The second type is the regression object detection algorithm, which provides for algorithms like SSD and YOLO. Traditional methods, also known as non-neural methods, use hand-crafted feature extraction algorithms such as HOG, SIFT, Haar wavelets, and custom-designed filters based on expert domain knowledge. While deep learning methods are often praised for their success in real-time object detection, traditional feature extraction algorithms still possess unique advantages in solving these problems. For instance, hand-crafted feature algorithms can run efficiently on essential CPU platforms, whereas deep neural networks often require powerful GPUs or dedicated hardware acceleration.
Furthermore, hand-crafted feature algorithms rely heavily on expert domain knowledge or features of the object in question, making them less dependent on large labeled training datasets that can be costly to scale up. This thesis proposes a system for real-time object detection in embedded devices that use stationary cameras and lightweight algorithms. The system utilizes the stationary camera's background characteristics, which remain almost constant. To detect objects, background extraction techniques are employed, and Zifian Estimation is used to detect moving objects to push into the object detection module.
The HOG- SVM algorithm is also used for calculations, which are parallel at various scales to avoid missing objects. The algorithm employs quantized values to reduce the number of complex computations in the system. Overall, these suggestions help to improve the efficiency of the system. The proposed system architecture involves using a sequence of images as input, which will be pre-processed using Zipfian Estimation Techniques.
This technique is useful when dealing with dynamic objects like humans, as it calculates only those within the frame. The areas where motion is detected are then directed to the Object Detection block, which comprises a Scale Generator module, six HOG-SVM computation modules running in parallel in a multithread architecture, and an NMS algorithm module. The Scale Generator and multithread HOG-SVM computation modules are essential to reducing processing time when scaling the input image to detect all humans in multiscale. The results are then merged with the original image for viewing, storage, or transmission to a server for further processing.
The system being proposed is intended for deployment on FPGA. However, due to time constraints in this thesis, the proposed system will be halted on software at the simulation stage. The proposed system underwent testing on a C++ software platform on a computer with an Intel® Core™ i5-1035G4 processor. Input video sources were obtained from the MOT15 dataset.
The evaluation of the system was based on speed and accuracy. The speed of the system was relatively good, but the accuracy was low. This could be attributed to various factors such as the complexity of the input images, limitations of the SVM training set, and not selecting suitable thresholds for each step like Zipfian Estimation, HOG-SVM, or NMS. Additionally, quantized values to optimize speed in steps contributed to the low accuracy results.
However, the bright spot is that the low complexity system has been appropriately optimized for future hardware platforms such as Xilinx's FPGA platform. Additionally, the above problems can be entirely improved by pushing our system into a specific situation, giving it an appropriate training set of SVM weights, and performing many corresponding experiments in many datasets with similar conditions with proposal situations to choose thresholds suitable for each step, from these things to complete the system.