Doctoral Dissertation Feature Extraction and Human-Human Interaction Recognition for Video Surveillance NGUYEN, Thuy Ngoc Supervisor: Associate Professor Atsuo Yoshitaka School of Information Science Japan Advanced Institute of Science and Technology June, 2016 Abstract Human interaction recognition has been widely studied because it has great scientific importance and many potential practical applications. However, this problem is very challenging especially in realistic environments where background is dynamic and has varying lighting conditions. This dissertation addresses human activity recognition, espe- cially human-human interactions in realistic video material, such as movies, surveillance videos. For classification problem, most existing methods rely on either spatio-temporal local features (ie.
SIFT) or human poses, or human joints to model human interactions. As a result, they are not fully unsupervised processes because they require either hand- designed features or human detection results. Motivated by the recent success of deep learning networks, we investigate a three-layer convolutional network which uses the Inde- pendent Subspace Analysis (ISA) algorithm to learn hierarchical invariant features. The ISA algorithm is a generalization of the Independent Component Analysis (ICA), which is very well-known in natural image statistics.
Compared to the ICA algorithm, the most notable advantage of the ISA is that it can learn features which are invariant to phase while being selective to orientation and frequency. However, the ISA algorithm becomes slow when applying it on video data. In order to solve this computational problem, we combine the idea of convolutional neural network with the ISA algorithm. Specifically, instead of training the ISA algorithm directly on raw video data, we first train it on small video blocks extracted by our procedure.
The obtained features are then convolved with larger video blocks. The outputs of this convolution step are fed into the next layer, which is implemented by another ISA algorithm. This organization enables the three- layer convolutional ISA network to learn hierarchical invariant features. Furthermore, we introduce a pooling layer to reduce the contributions of features learned in lower lay- ers while still achieving translation invariant.
Using the invariant features learned by the three-layer convolutional ISA network, we build a bag-of-features representation for videos. Finally, we apply Support Vector Machine (SVM) to classify human interactions. For temporal localization, we slide temporal detection windows with different durations over a continuous video sequence with a stride of 10 frames. For each temporal window, our convolutional ISA network extracts hierarchical invariant features on a dense grid.
After scoring the temporal detection windows, a non-maximum suppression is applied to enforce that non of the retained windows are overlapping. In two cases, we conduced thorough experiments on realistic videos from challeng- ing benchmarks used by activity recognition community. We show that our three-layer convolutional ISA network is effective to represent complex activities such as human in- teractions in realistic environments. Besides, we believe that our temporal localization method is the first work which reports experimental results on the continuous video se- quences of human interactions.
Although temporal localization results are insufficient for real applications, it is a first step for further research in localization of human interactions. Keywords: Temporal localization, Classification, Independent subspace analysis, Human-human interactions, Convolutional neural network, Pooling. 1 Contents Abstract 1 Introduction 1.3 Challenges with human activity recognition.1 Intra-class and inter-class variatlons. gà kg va 1.
HH ng g va 2 Related Work 11 2.1 Local feature methods .1 Spatio-temporal features.2 Global representation methods.1 Shape mask and silhouette based methods .2 Optical flow and shape based methods .3 Body part based methods .1 Overview of activity recognition datasets .2 Datasets for experimental evaluation. The UT-Interaction dataset. The Hollywood2 dataset. ili 4 Interaction Recognition using Hierarchical Invariant Features 31 4.2 Independent Subspace Analysis (ISA) for image data .3 Three-layer convolutional ISA network .1 Video block extraction .2 Hierarchical invariant features.
cu ng ng kg kg va 44 4.1 Bag-of-features representation .2 Support vector machine .000 eee 46 45 Experimental results .3 Analysis of parameter settingS.6 Summary of interaction recognition. co 55 5 Interaction Temporal Localization based on Sliding Window Approach 56 5. gà gà kg va 56 5.2_ Interaction localization based on temporal sliding window.1 Temporal sliding window. Extraction of hierarchical invariant features .4 Non-maximum suppression.4 Summary of interaction temporal localization.
63 6 Conclusion and Perspective 65 6. kg và ga 65 6. cà gà gà kg va 66 Publications 67 iv List of Figures 1.1 Sample actions in the KTH dataset [84|.2 Sample frames of realistic video data - UCF Sports [78] .1 Harris3D interest points for the motion of the legs of a walking person. Left image is a 3D plot with a threshold level surface of a leg pattern (upside down) and the detected points.
Right image shows the detected interest points [52].2 Feature trajectories are obtained by detecting and tracking interest points. Trajectories are clustered into a library of trajectons [63].3 Overview of the dense trajectories [103].4 The sample shape masks for the backhand stroke activity from the tennis activity [110]. GY Mi gp.5 MEI and MHI representation for two sample movements.6 Space-time shapes of ‘jumping-jack’, ‘walking’, ‘running’ [6].7 Motion descriptor using optical flow (a) Original image (b) Optical flow F,(c) Separate the z and y components of optical flow vectors F,, Fy (d) Half-wave rectification of each component to produce 4 separate channels Fy, F>, F7, and F_, (e) Final blurry motion channels Fb}, Fb;, Fb;, and Foy (23).8 Examples of annotated poselets [75].1 Example snapshots of six classes of two-person interactions in Set 1.2 Example snapshots of six classes of two-person interactions in Set 2.3 Sample frames for the Hollywood2 action dataset. Three samples are given for each of the twelve action classes.4 Sample frames for the Hollywood2 action dataset.
Three samples are given for each of the twelve action classes (continue from Figure 3.1 The neural network architecture of an ISA network. The blue and red bubbles represent units in the first and second layer respectively. In this figure, the dimension of subspace is 2: each red bubble looks at 2 blue bubbles.2 Six groups of filters (produced by W) learned from the ISA algorithm when trained on images patches. Each row contains three groups of filters.
The grey-scale value of a pixel means the value of coefficient (weight) at that pixel. Grey pixels mean zero coefficients. Each group of two filters represents a feature subspace (or a neuron in the second layer).3 Responses ƒ; of a feature subspace i. (a) Two underlying filters in a feature subspace.
(b) Effect of varying frequency. (c) Effect of varying orientation. (d) Effect of varying phase. The response values are normalized so that the maximum response is equaltol.4 Graphical depiction of applying the ISA algorithm to video data.5 Representation of extracted video blocks.
Each video block is a sequence of frames. The red rectangle corresponds to a complete video block. The video blocks in (a) are randomly extracted from videos without applying frame differencing. The ones in (b) are obtained as a result of applying frame differencing.6 An illustration of the convolution step in the three-layer convolutional ISA network.7 Typical filters learned by the convolutional ISA network on 16 x 16 x 10 video blocks.
Each row is a filter in 3D (drawn from a row of the matrix W). It can be clearly seen that frames in a row resemble each other closely while their differences are still recognizable.8 Confusion matrices of per-clip classification results on Set 1 and Set 2 of the UT-Interaction dataset. Horizontal rows of the matrices represent ground truths, and vertical columns are predictions.9 Confusion matrices for the UT-Interaction dataset of two previous methods (a) Waltisberg et al. [102], (b)Vahdat et al.10 Recognition results on the UT-Interaction dataset with changing the num- ber of extracted video blocks.1 The procedure of our localization method on continuous video sequences.2 Several wrong localization cases of our temporal localization method for the UT-Interaction dataset.
ee vi List of Tables 3.1 Development of activity recognition datasets.2 Summary of the statistics of the UT-Interaction dataset.3 Description of the UT-Interaction dataset. ‘Pedestrian’ indicates whether the scene of video sequence contains irrelevant pedestrians. ‘Simultaneous interactions’ describes whether the video sequence consists of two pairs of simultaneous executions.4 Summary of the statistics of the Hollywood2 dataset .5 Distributions of class instances of the Hollywood2 dataset.1 Classification performance for the UT-Interaction dataset.2 Classification performance per action class for the Hollywood2 dataset.3 Performance comparison (in terms of recall) for the UT Interaction dataset.4 Performance comparison (in terms of mAP) for the Hollywood? dataset.9 Performance comparison with and without applying frame differencing in video block extraction .6 Recognition results on the UT-Interaction dataset in different choices of block size. ww và à kg k k TT và 5.1 Duration (in frames) per interaction class of Set 1 of the UT-Interaction dataset.2 Duration (in frames) per interaction class of Set 2 of the UT-Interaction dataset.3 Localization results on the Set 1 of the UT-Interaction dataset.4 Localization results on the Set 2 of the UT-Interaction dataset.
vil Chapter 1 Introduction 1.1 Introduction Videos help to improve quality of contents and communication by combining visual, au- dio, and textual information in multiple data streams. Over the past years, with the rapid development of high technology and faster internet access, video data has become superfluous both in off-line storage and on the internet. Specifically, electronic devices such as computers, tablets, mobile phones are used almost everywhere, and people can record, store, and share videos easily. According to statistics in 2016 [20] from the most popular sharing website - YouTube: e 5 billion videos are watched on the YouTube every single day.
e 300 hours of video are uploaded to it every minute. e Total number of people who use YouTube is 1. This shows that videos as well as video cameras have become an inseparable part of our lives. With a fast growing number of videos and with such widespread popularity of watching videos and movies on the Internet, there is an urgent need for advanced video analysis techniques that can systematically interpret and understand the semantics of video contents.
Automatic understanding the content of a video is a long-standing goal of computer vision and it is interesting to identify which objects are the most important in videos. Ivan Laptev [51] performed an experiment and found out that about 35% of screen pix- els in movies, TV programs and YouTube videos belong to people. Besides, images on Flickr.com contain about 25% of person pixels. These numbers imply that the visual data we tend to produce, share and consume is strongly biased toward people.
However, the percentage of person pixels in a first-view wearable camera dataset [39] is only about 4%. This further indicates that the strong person bias in consumer videos is not natural and is created with the intention of video maker or editor. With the strong bias of video toward people, understanding the semantics of video contents requires the need of automatic methods that interpret person pixels in videos. There are several levels of interpreting person pixels which can lead to understanding of video contents including people detection, people tracking and analyzing their activi- ties.
People detection is the process of verifying the presence of a person (or people) in image sequences and identifying their positions precisely. People tracking is to determine the location of a moving person (or people) during a video sequence. Generally, people detection and tracking are closely related because people tracking usually starts with de- tecting people. Although people detection and tracking have a wide range of applications, these two processes can only answer simple questions like who are in a video and their trajectories.
However, the more interesting information is people behaviors, i. what are they doing in a video. Thus, the natural step is to recognize human activities occurred in videos. Human activity is defined as a collection of human movements with a particular se- mantic meaning.
Recognizing human activities in video is of high interest because of many applications such as entertainment, education, security and surveillance.