VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY COMPUTER ENGINEERING DEPARTMENT LE MINH PHUC BÙI HỮU TRÍ GRADUATION THESIS RESEARCH AND EVALUATE HARDWARE IMPLEMENTATION OF CNN MODEL NGHIEN CUU VA PHAT TRIEN MO HINH CNN TREN PHAN CUNG ENGINEER OF COMPUTER ENGINEERING HO CHi MINH CITY, 2021 VIETNAM NATION UNIVERSITY HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY COMPUTER ENGINEERING DEPARTMENT LE MINH PHUC- 17520903 BUI HUU TRI - 17521153 GRADUATION THESIS RESEARCH AND EVALUATE HARDWARE IMPLEMENTATION OF CNN MODEL NGHIEN CUU VA PHAT TRIEN MO HINH CNN TREN PHAN CUNG ENGINEER OF COMPUTER ENGINEERING INSTRUCTOR PhD. LAM DUC KHAI HO CHi MINH CITY, 2021 LIST OF THE COUNCIL TO PROTECT THE THESIS Graduation thesis grading council established under Decision no 62/QD-DHCNTT Dated 14/02/2022, of the Rector of the University of Information Technology. THANK YOU After studying at the University of Information Technology, we would like to send our sincere thanks to the teachers of Computer Engineering have taught us knowledge and invaluable experiences Especially, we are also sincere thanks to PhD. Lam Duc Khai for helping us and taking the time to guide and instruct us so that we can complete the graduation thesis.
Once again, we would like to sincerely thank the council for spending their time and effort to help us in the graduation thesis process. We apologize to everyone because of the mistakes, we hope teachers and you can ignore and forgive. 6-6 T TH HH ngư 3 I1 ác. ác Sà SH HH HH ướt 4 2.
Feed Forward Algorithm. Convolutional Neural NetWOrK. Padding and Stride. Fully Connected Layer.
ras ee snnensss sins cossscsvesessssesssseessssesocsssossocsssosssssseosssessecsssooss 24 2. CTOSS-En(TODV. ch HH HH Hết25 2. Backpropagation in Fully Connected Block.
Backpropagation in Convolutional Bloek.- - «5+ xkEkekE SH HH ưu34 2. SYSTEM ARCHITECTURE DESIGN AND IMPLEMENTATION. CNN Datapath IP architecture. TH HH HH HH HH HH it37 3.
Max Pooling module:. Arithmetic sub-module:. Floating-point adder:. Floating-point multipÏiCafiOn:.
Floating-point division:. Rock-paper-scissors dataset:. VGG-16 Overall architecture:. CNN Datapath IP:.
CONCLUSION AND DEVELOPMENT DIRECTIONS. CNN Datapath IP. Waveform and accuracy per epoch. Comparison with related articles (Testing mode only): .Ă Sàn HH HH Hit79 4.
System RÑ€SOUTCCS. Comparison with related articles. _ What We Gained, Limitation and Direction of Development:. What We Gained: 0.
cece ees necsesesesneeaeseseesesesesesneeseseesseeneaeee 82 4. Direction of Development LIST OF FIGURES Figure 2.1: Basic Neural Network .- - - tình hết Figure 2.2: Feed-forward algorithm Figure 2.4: A complete convolutional neural network [6].5: Kernel in the field of Computer Vision [9] Figure 2.6: Simulation of Convolution [6].7: Padding = 1 for input matrix.8: Red box - Kernel, Black box — Input.9: Convolve valid padding Figure 2.10: Convolve same padding .11: Convolve full padding Figure 2.12: Stride = 1 for data matrix, the yellow tiles are kernel core.16: Fully Connected Layer Figure 2.17: Graph of Sigmoid function.18: Graph of Tanh function.19: Graph of ReLU function.20: Graph of Softmax function.21: Neural Network example .22: Convolutional example Figure 2.23: Mask of Pooling .25: VGG-16 Structure Figure 2.1: CNN Datapath IP .2: CNN Datapath IP details .3: Convolve module block diagram.4: ReLU module block diagram Figure 3.5: Max Pooling module block diagram.6: Flatten perceptron module block diagram .7: Softmax module block diagram .8: CFED module block diagram.9: Flatten perceptron module block diagram .10: CECC module block diagram Figure 3.11: CECC module operation.12: Update weight module operation.13: Update kernel module operation Figure 3.14: Convolve valid module operation .15: Floating-point adder module block diagram Figure 3.16: Floating-point multiplication module block diagram.17: Floating-point division module block diagram.18: Exponential module block diagram Figure 3.19: Rock-paper-scissors dataset [8] Figure 3.21: VGG-16 architecture Figure 3.22: Quantization module Figure 3.23: Mem block example.24: Mem block signal.25: Testbench environment block diagram Figure 3.26: Test flow Figure 3.27: Testbench environment block diagram Figure 3.28: Test flow Figure 4.1: Mixing testing mode and learning mode Figure 4.4: Waveform of VGG16. LIST OF TABLES *Table 3.1: CNN Datapath IP IO information.2: Convolve module IO information.3: ReLU module IO information.4: Max Pooling module IO information.5: Max Pooling module IO information.6: Softmax module IO information.7: CFED module IO information.8: CDCP module IO information.9: CECC module IO information.10: Update weight module IO information.11: Assuming of LUT consumption if implement with Feed-forward Convolve architecture.12: Assuming of LUT consumption if implement with 3.13: Update kernel module IO information.14: Floating-point adder IO information.15: Floating-point multiplication IO information.16: Floating-point division IO information.17: Exponential IO information.1: Accuracy per epoch started from epoch 0.2: Accuracy per epoch started from epoch 12.3: Accuracy per epoch started from epoch 24.4: The error of Convolve, Pooling, and FC.5: The error of Softmax.6: The error of CFED, CECC, and CDCP.7: The error of Update weight, Update kernel.8: Implementation result of CNN Datapath IP.9: Implementation result of Feed-forward sub-modules.10: Implementation result of Backpropagation sub-modules (P1).11: Implementation result of Backpropagation sub-modules (P2).12: Comparison between [1] and our work.13: Comparison between RTL accuracy and Matlab accuracy.14: Implementation result of VGG-16 sub-modules.15: Implementation result of VGG-16.17: Comparison between [2] and our work *Table 4.18: Errorincrease per epoch while updating parameters (U#: update 1). 83 LIST OF ACRONYMS DSP: Digital Signal Processing FF: Flip-Flop FIFO: First in First Out FPGA: Field Programmable Gate Array GPU: Graphics Processing Unit HDL: Hardware Description language IP: Intellectual Property ISE: Integrated Synthesis Environment LUT: Look-Up Table RAM: Read Access Memory ReLU: Rectified Linear Units RTL: Register-transfer Level SRAM: Static Random-access Memory SUMMARY OF THESIS In recent years, the Deep Learning models have been interested in many scientists to participate in research, notably the Neural Network model as a good candidate to solve problems such as recognizing object processing.
Nowadays many companies have produced the AI chip by using the cloud for the training process, and it will consume a huge number of internal chips when it develops in the future. In response to this problem, developing an IP for learning will reduce the cost of production. We propose an architecture, design of Convolutional Neural Network architecture that can learn, test and VGG16 using quantization to reduce resources and get better accuracy. Introduction In recent years, along with the explosion of the 4.0 revolution, Machine Learning plays an important role in human life.
We can find it in many devices around us and it also is an important thing to build many future devices such as an automated car. CNN is one of the most popular models of machine learning, it has many architectures and uses, but image classification is one of the most basic and commonly used. To implement machine learning, nowadays many companies have produced the AI chip, with a specific CNN architecture such as YOLO, GoogleNet, VGG-16, etc. Mostly the AI chip nowadays is implemented with the Feed-forward operation only, and the parameter will be trained via Cloud.
With the growth of AI as well as human’s needs, the amount of data also growth, and the needs of the server, ethernet to perform the AI training also increased, resulting in a high cost in server, ethernet chip. To solve the above problem, our team is implementing a CNN IP capable of doing both the learning and applying. With a CNN IP capable of learning and applying operation, the pressure on the server, ethernet chip production. Besides, to reduce the area consumption and increase performance on AI chips, our team also implemented a VGG-16 IP based on the Quantization method.
With the Quantization method, the AI chip arithmetic operation will use low-bit width arithmetic such as 8-bit integer adder, multiplication, and shifter. With low-bit width arithmetic, result in a significant reduction of area and the increment of speed. Project’s goal There are 2 main goals in our project: The first goal, design a CNN model on hardware capable of learning and testing. There are two parts to this goal, one is making a Feed-forward structure to perform a part of learning and real-time testing, and the Backpropagation structure to perform 13 the learning, compatible with the Feed-forward, using floating-point value.
The CNN model will be experimented with both learning and testing by using the MNIST dataset. The second goal, design a VGG-16 model on hardware capable of doing real-time detection with high accuracy and performance. In this goal, we will use the quantization method, to achieve good accuracy and performance with an only 8-bit integer value. Neural Network A neural network is built based on a biological neural network inspired by the human brain.
It consists of neurons that connect and process information by passing information along with connections and calculating values at the neurons. Each neuron will have input data, which will be changed by the inputs and weights. A layer is many neural nodes that are the same input but have different weights (Figure 2. After going through all the layers, the input data will be processed and then combined in the final layer to create a prediction.
The neural network learns by generating difference errors between the network's predictions and the desired value and then uses those errors to update the weights and biases again for more accuracy.1: Basic Neural Network 2. Feed Forward Algorithm @ Bias unit Wo — INPUT NEURON OUTPUT Figure 2.2: Feed-forward algorithm In Figure 2.2, the inputs x are being fed to the red neuron along with w in their path. The input value of the red Neuron is called Val with the formula: 15 val = 3?-+w¡x¡ + b (2.1) In which, the main b value is w0 in Figure 2.2, which is called the bias weight. Each neuron when producing output will have to go through an activation function, the formula is as shown in Figure 2.
Convolutional Neural Network A convolutional neural network is a commonly used network in image processing and classification, including two large blocks, feature extraction blocks (including convolutional blocks, pooling blocks), and classification blocks.3 shows an example of a CNN model to classify 5 classes: car, truck, airplane, ship, horse. Besides that, Figure 2.4 illustrates the shape of the image when it goes through each convolutional block. car truck SÏfplane Ship horse Figure 2.3: CNN example [5] 16 t3 f4 Fully-Connected Fully-Connected Neural Network Neural Network Conv_1 Conv_2 ReLU activation Convolution Convolution | —*®—x xsl ke xế Max-Pooling exe5) Karel Max-Pooling (with valid padding (2x2) valid padding (2x2) dong A % A AL | ® ®o a. ie poe INPUT n1 channels, n1 channels n2channels n2channels, 2 ï (28x28 x1) (24x24 xn1) (12x12xn1) (8x8xn2) — (4x4xn2) @ om n8 units Figure 2.4: A complete convolutional neural network [6] 2.
Convolutional layer A multiplication matrix (or W matrix) is a matrix of size KxK where K is an odd number. This matrix is used to get the features of an image through convolution, generating feature matrices of the input image data depending on the type of Kernel matrix used. The feature matrix is also known as the feature map. Note that the output feature map will have the width size equal to (input width — kernel width + 1).
The performance of the convolution operation also depends on the following factors: kernel matrix size, padding, stride. Convolution formula (Figure 2.6) for each element to get the feature matrix: yị=sum(A@K) (2.3) In which A is the image input, K is the kernel matrix applied to calculate the convolve. yij is a single-pixel output of the convolve. 17 Operation Kernel w 000 Identity 010 0-1 0 Sharpen 15 -1 Box blur 1 (ales) 9 Figure 2.5: Kernel in the field of Computer Vision [9] : Convolved Feature Figure 2.6: Simulation of Convolution [6] In Figure 2.5, there is the operation of convolutional and several common kernel matrixes that we can use in computer vision.
Padding and Stride 2. Padding Padding is simply a process of adding layers of zeros to our input images. 18 We call padding = n where n > 0 is the number of layers of zero added outside the matrix.7: Padding = 1 for input matrix Padding (example in Figure 2.