BỘ GIÁO DỤC VÀ ĐÀO TẠO TRƯỜNG ĐẠI HỌC BÁCH KHOA HÀ NỘI --------------------------------------- Khổng Văn Minh KẾT HỢP ĐẶC TRƯNG DIỆN MẠO VÀ CHUYỂN ĐỘNG TRONG BIỂU DIỄN HOẠT ĐỘNG CỦA NGƯỜI SỬ DỤNG MẠNG NƠ RON TÍCH CHẬP Chuyên ngành : Hệ thống thông tin LUẬN VĂN THẠC SĨ KHOA HỌC HỆ THỐNG THÔNG TIN NGƯỜI HƯỚNG DẪN KHOA HỌC : TS. Trần Thị Thanh Hải Hà Nội – Năm 2018 17057204899671000000 MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY --------------------------------------- KHONG VAN MINH COMBINATION OF APPEARANCE AND MOTION INFORMATION IN HUMAN ACTION REPRESENTATION USING CONVOLUTIONAL NEURAL NETWORK FIELD OF STUDY : INFORMATION SYSTEM MASTER’S THESIS IN INFORMATION SYSTEM SUPERVISOR: PhD: Tran Thi Thanh Hai HANOI – 2018 SĐH.BM11 CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM Độc lập – Tự do – Hạnh phúc BẢN XÁC NHẬN CHỈNH SỬA LUẬN VĂN THẠC SĨ Họ và tên tác giả luận văn : Khổng Văn Minh Đề tài luận văn: Kết hợp đặc trưng diện mạo và chuyển động trong biểu diễn hoạt động của người sử dụng mạng nơ ron tích chập Chuyên ngành: Hệ thống thông tin Mã số SV: CBC17021 Tác giả, Người hướng dẫn khoa học và Hội đồng chấm luận văn xác nhận tác giả đã sửa chữa, bổ sung luận văn theo biên bản họp Hội đồng ngày….………… với các nội dung sau: ……………………………………………………………………………………………………. Ngày tháng năm Giáo viên hướng dẫn Tác giả luận văn CHỦ TỊCH HỘI ĐỒNG Abstract In this thesis, I focus on solving the action recognition problem in video or a stack of con- secutive frames. This problem plays an important role in surveillance systems that are very popular nowadays.
There are two main solutions to solve this problem: using hand-crafted features or using learned features using deep learning. Both of the solutions have pros and cons and the solution that I study belongs to the secondategory. Recently, advanced tech- niques relying on convolutional neural networks produced impressive improvement com- pared to traditional handcrafted features based techniques. Besides, literature researches also showed that the use of different streams of data will help to increase recognition per- formance.
This paper proposes a method that exploits both RGB and optical flow for human action recognition. Specifically, we deploy a two stream convolutional neural network that takes RGB and optical flow computed from RGB stream as inputs. Each stream has ar- chitecture of an existing 3D convolutional neural network (C3D) which has been shown to be compact but efficient for the task of action recognition from video. Each stream works independently then is combined by early fusion or late fusion to output the recognition results.
We show that the proposed two-stream 3D convolutional neural network (2stream C3D) outperforms one stream C3D on two benchmark datasets UCF101 (from 82.87%) and CMDFALL (from 65. 1 Acknowledgments Firstly, I would like to express my deep gratitude to my supervisor PhD. Tran Thi Thanh Hai for supporting my research direction, which allowed me to explore new ideas in the field of computer vision and machine learning. I would like to thank for her supervision, encouragement, motivation, and support and her guidance helped me throughout the re- search work and in writing of the thesis.
I would like to acknowledge the International Research Institute MICA, HUST for pro- viding me the great research environment. I wish to express my gratitude to the teachers in Computer vision department, MICA for giving me the opportunity to work and acquire great research experience. I would like to acknowledge the School of Information and Communication Technology for providing me the knowledge and the opportunity to study. I would like to thank my friends for supporting me in my study.
Last but not least, I would like to convey my deepest gratitude to my family for their supports, and sacrifices during my studies. 2 Contents 1 Introduction to Human Action Recognition 9 1.1 Human Action Recognition problem .2 Overview of human action recognition approach .1 Hand crafted feature based methods .2 Deep learning based methods .3 Purpose of thesis. 13 2 State-of-the-art on HAR using CNN 15 2.1 Introduction to Convolutional Neural Networks .2 2D Convolutional Neural Networks .3 3D Convolutional Neural Networks .4 Multistream Convolutional Neural Networks. 20 3 Proposed method for HAR using multistream C3D 23 3.3 Optical Flow Stream .4 Fusion of multistream 3D CNN .1 Pros and Cons.
43 4 List of Figures 1-1 Human Action Recognition Problem. 10 1-2 Human Action Recognition phases. 11 1-3 Hand-crafted feature based method for Human Action Recognition. 12 1-4 Deep learning method for Human Action Recognition problem.
13 2-1 Main layers in Convolutional Neural Networks. 15 2-2 Fusion techniques used in [1]. 19 2-4 Two stream architecture for Human Action Recognition in [2]. 21 3-1 General framework for human action recognition.
24 3-2 Early fusion method by concatenate two L2-normalization feature vectors. 26 3-3 Late fusion by averaging class score. 27 4-1 The class labels in UCF101 dataset. 29 4-2 The class labels in HMDB51 dataset.
30 4-3 Experiment steps for each dataset. 30 4-4 The step using C3D for experiment. 32 4-5 C3D clip and video prediction. 35 4-6 Confusion matrix of two stream on UCF101.
36 4-7 Confusion matrix of two stream on HMBD51. 36 4-8 Confusion matrix of two stream on CMDFALL. 37 4-9 In HMDB51, the most confused action in the RGB stream is swing base- ball. 60% of its videos are confused with throw.
39 5 4-10 Most benefit classes in UCF101 when combining compared to RGB stream 39 4-11 Most benefit classes in HMDB51 when combining compared to RGB stream 40 4-12 Most benefit classes in HMDB51 when combining compared to RGB stream 40 4-13 Classes of UCF101 in which RGB stream perform better. 40 4-14 Classes of UCF101 in which Flow stream perform better. 41 4-15 Classes of HMDB51 in which RGB stream perform better. 41 4-16 Classes of HMDB51 in which Flow stream perform better.
41 4-17 Classes of CMDFALL in which RGB stream perform better. 41 4-18 Classes of CMDFALL in which Flow stream perform better. 42 6 Acronyms 3DCNN 3D Convolutional Neural Networks. 1, 19 CNN Convolutional Neural Networks.
1, 15, 17, 20 HAR Human Action Recognition. 1, 9, 23 HOG Histogram of Gradients. 12 MBH Motion boundary histograms. 12 SIFT Scale-invariant feature transform.
12 7 List of Tables 2.1 Result of fusion techniques on the 200,000 videos of the Sport1M test set. Hit@k indicate the fraction of test samples that contained a least one of the ground truth labels in the top k predictions [1].2 C3D results on different tasks .3 Two-stream architecture mean accuracy (%) on UCF101 and HMDB51 dataset .1 Class tree of CMDFALL dataset .2 Accuracy of action recognition on single and multiple streams C3D (%) .3 Comparision result on two popular benchmark datasets (%). 37 8 Chapter 1 Introduction to Human Action Recognition 1.1 Human Action Recognition problem Human action recognition is an important topic in computer vision domain. It has many applications such as: surveillance system in hospital, abnormal activity detection in build- ing (bank, aeroport, hotel) or in human machine interaction.
There are various types of human activities. Depending on their complexity, we can categorize human activities into four different levels: gestures, actions, interactions, and group activities. ∙ Gestures are elementary movements of a person"s body part, and are atomic compo- nents describing the meaningful motion of a person. Example: "stretching an arm", "raising a leg",.
∙ Actions are single person activities that may be composed of multiple gestures orga- nized temporally, such as: "walking", "waving", and "punching". ∙ Interactions are human activities that involve two or more persons and/or objects. For example, "two person fighting" is an interaction between two humans, "drinking water" is an interaction between human and object. ∙ Group of activities are the activities performed by conceptual groups composed of 9 multiple persons and/or objects.
Example: "A group of persons marching",. Figure 1-1: Human Action Recognition Problem In this thesis, we focus on the human action recognition. The problem of human action recognition can be defined as below. ∙ Input: A video or a sequence of consecutive frames that contain a human action.
∙ Output: Label of the action that that belongs to one of the predefined classes. Human action recognition is a challenge for researchers in computer vision domain because of noisy background, viewpoint changes, and variety in performing action of each person. Figure 1-1 illustrates the human action recognition problem. Key components of a visual recognition system Figure 1-2 illustrate the two phases of a recognition system.
∙ Training: Learning from the training dataset to obtain the parameters of the recogni- tion model. ∙ Recognition: Use the learned model from training phase to recognize new data. Each phase in the system has the main components as below: ∙ Preprocessing data: Convert data to the form that are compatible for the model 10 Figure 1-2: Human Action Recognition phases ∙ Feature extraction: From the preprocessed data, extract the suitable features for rep- resenting the human action. The features can be obtained by hand crafted or deep learning techniques.
∙ Classification: Use the features extracted from previous step to create the input for the training or predicting. ∙ Recognition: The new data is input through the step of preprocessing, feature extrac- tion, then using the trained classifier for predicting the label. 11 Figure 1-3: Hand-crafted feature based method for Human Action Recognition 1.2 Overview of human action recognition approach 1.1 Hand crafted feature based methods In this approach, human actions are represented by features that are manually designed by high experience researchers. Once features are extracted, they are inputs to a generic trainable classifier for action recognition.
The building blocks for hand-crafted feature- based approach is illustrated in the figure 1-3: ∙ Feature extraction: Takes input as image or video pixel and output the features for that image or video. ∙ Classification: A classifier that takes the feature as input and provides the output as class label. There are many types of handcrafted features designed by experts to solve the human action recognition problem. Many classical image features have been generalized to videos, e.
Among local space-time features, dense trajectories have been shown to perform best on variety of datasets. The main idea is to densely sample feature points in each frame, and track them in the video based on optical flow. Multiple descriptors are computed along the trajectories of feature points to capture shape, appearance and motion information. Motion boundary histograms (MBH) give the best results among these descriptors.
The idea of dense trajectories has extended by the work of Wang and Schmid [3] to improve of performance by considering the camera motion and achieved state-of-the- art in hand-crafted feature. Despite its good performance, this method is computationally intensive. 12 Figure 1-4: Deep learning method for Human Action Recognition problem 1.2 Deep learning based methods On the other hand, a learning-based representation approach, specifically, deep learning uses computational models with multiple processing layers to learn multiple levels of ab- straction from data. This learning encompasses a set of methods that enable the machine to process the data in raw form and automatically transform it into a suitable representation needed for classification.
This is what we call trainable feature extractors. This transfor- mation process is handled at different layers. These layers are learned from raw data using general purpose learning procedure which does not need to be designed manually by ex- perts. The performance of the human action recognition methods mainly depends on the appropriate and efficient representation of data.
Recently, deep learning achieved very good result on image-based task [4]. This result inspires researchers to extend it into video classification specially to solve the human action recognition problem.