VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY Kieu Hai Dang Weighted PCA Regression To Recover Missing Markers In Human Motion Data MASTER THESIS Major: Computer Science HANOI - 2021i VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY Kieu Hai Dang Weighted PCA Regression To Recover Missing Markers In Human Motion Data Major: Computer Science Code: 8480101.01 Supervisor: Assoc Prof. Le Thanh Ha HANOI - 2021 ABSTRACT “Missing markers problem”, that is, missing markers during a motion capture session, has been raised for many years in Motion Capture field. Some algorithms have been proposed but they still show some limitations. Recent approaches used Principal Component Analysis (PCA) to create a single basis system from the un- missing frames in the motion sequence.
However, we are aware that those approaches of PCA have limitations in case of missing multiple gaps with a very long sequence. With the rising amount of Motion data, we question how could we use a part of the available Mocapdata to improve the performance of the current state of art algorithms. In this study, we aimed to present methods that could utilize the available Mocap dataset to enhance the performance of the current state-of-the-art algorithm. We first applied PCA into a collection of samples extracted from the Mocap dataset to gain a set of basis systems.
Then, We further used least square method to compute a weight vector for the basis systems. Finally, interpolation is the synthesis of the basis systems with their weight. To test the performance, we analyze our algorithms and compare with original PCA approaches in 2 scenarios of missing gaps in famous Mocap datasets (Carnegie Mellon University Dataset and Motion Capture Database HDM05). Each experiment is conducted over 50 times to guarantee effectiveness.
Our final results show that my methods outperform the original PCA approaches. Index Term: Missing markers problem, MoCap data, principle component analysis i ACKNOWLEDGEMENTS First of all, I would like to express special gratitude to my supervisor – Assoc Prof. Le Thanh Ha for his enthusiastic instructions, technical explanations as well as advice during this project. I have received a lot of detailed guidance from my supervisor when I faced problems.
I could haven’t been completed my thesis without the instruction of those from email and weekly meetings. I also want to give sincere thanks to Dr. Hong Chuan Yu (Bournemouth University, UK) for the instructions as well as the background knowledge for this thesis and the guidance when I worked at Bournemouth University. I would like to also thank my colleagues in Human Machine Interaction Lab for their support.
Last but not least, I want to thank my family and all of my friends for their motivation and support as well. They stand by and inspire me whenever I face a tough time. ii DECLARATION I hereby declare that the research and experimental results presented in the thesis are my own and under the guidance of Assoc. Le Thanh Ha and Dr.
Hong Chuan Yu. I confirm that the work has never been used as any other master thesis or other degrees of any university. iii TABLE OF CONTENTS ABSTRACT. iii TABLE OF CONTENTS.iv LIST OF FIGURES .vi LIST OF TABLES.
viii TABLE OF ABBREVIATIONS. Contributions and Thesis’s layout. Low-rank decomposition techniques. Model-based methods.
Locally Weighted PCA. Missing a single joint. Missing multiple markers. 32 THESIS-RELATED PUBLICATIONS.
39 v LIST OF FIGURES Figure 1: Human in Xsens suit and motion analysis .2 Figure 2: Joints in Mocap dataset .3 Figure 3: Examples of a motion sequence with 1 missing marker. The missing marker has coordinate (0,0) is indicated in black circle.5 Figure 4: Block diagram of the Tits method [9], probabilistic averaging method of several recovery models .7 Figure 5: Peng et al. [28] nonegative matrix factorization pipeline .9 Figure 6: Gloersen's method of using PCA to recovery missing gap .11 Figure 7: : Illustration of bi-directional attention network proposed by Qiongjie Cui [31] .12 Figure 8: Holden et al. [32] states the comparision of various techniques in his work .12 Figure 9: Logical flow of proposed approaches.13 Figure 10: Example of missing gaps in a testing sample.
YELLOW indicates gaps.22 Figure 11: Comparison of the reconstruction for missing single gap at the LKNE marker in the CMU dataset. Dotted black lines represent the original trajectory of missing marker, black lines represent the remaining trajectory in the LKNE marker, red lines represent our method reconstruction.25 Figure 12: Comparison of the reconstruction for missing single gap at LSHO and LWA marker in CMU dataset. Dotted black lines represent the original trajectory of vi the missing marker, black lines represent the remaining trajectory in the LWA marker, the red lines represent our method reconstruction.26 Figure 13: Boxplot of reconstruction error for missing 3, 6, 9 markers in a testing sample using CMU Dataset.28 Figure 14: Example reconstruction for missing multiple gaps at the same time frame in the right ankle marker.29 Figure 15: Performance stability along with the location of missing gap Reconstruction errors of our methods at missing single marker case for LKNE, LSHO in CMU and HDM dataset.30 Figure 16: Example of missing the marker LWA in motion sequence in the CMU data .33 Figure 17: LWPCA's reconstruction.33 vii LIST OF TABLES Table 4-1: Mean reconstruction error for missing a single gap .24 Table 4-2: Average reconstruction error for missing multiple gaps .27 Table 4-3: Running time per sample (s).31 viii TABLE OF ABBREVIATIONS Abbreviation Explanation PCA Principal component analysis SVD Singular value decomposition CMU Carnegie Mellon University Dataset HDM Motion capture database HDM05 LSHO Left shoulder joint LKNE Left knee joint LWA Left wrist anterior MoCap Motion capture database LDS Linear dynamic systems HMM Hidden Markov Model PMA Probabilistic model averaging model MSE Mean sqaure error NMF Nonnegative Matrix Factorization WLS Weighted least squares SPCA Stacked PCA WPCA Weighted PCA LWPC Locally weighted PCA 1 CHAPTER 1. Introduction Motion capture (MoCap) technology is widely used in our daily lives, spanning from clinical applications to sport coaching, movie visual effects creation, computer animation [1–4], and VR/AR, such as the LIDAR sensor on the iPad.
However, due to the human body's highly articulated structure and motion complexity, human motion tracking remains difficult and self-occlusion in body parts is one of the most well-known issues. With the availability of multiple camera systems, there are still missing gaps in the motion sequence. For example, twist and rotate body actions in sporting and dancing exercises usually create movements at a high degree of freedom. Figure 1: Human in Xsens suit and motion analysis 2 Moreover, some markers could be disappeared in multiple consecutive frames or be missed throughout in some extreme scenarios.
As the consequence, the productivity of the current motion analysers reduces significantly. Gaps in motion datasets, which is called the “Missing Marker Problem” [5]. Although there are some commercial software products available that can provide powerful tools for aiding in the clean-up of MoCap data [6], it can still often take several hours per capture and is almost always the most expensive and time-consuming part of the pipeline. Figure 2: Joints in Mocap dataset This paper focuses on the “missing marker problem” where a gap of a motion sequence in human motion could be randomly missed in an arbitrary marker.
The methods comprise linear, spline interpolation is usually used to fill the gap. However, 3 those methods are restricted to the number of the missing markers as well as the number of consecutive missing frames [5]. Xia Guiyu et al. in “Nonlinear low-rank matrix completion for human motion recovery” [8] proposed methods of low-rank matrix completion which exploit the sparsity of motion data.
However, there are existing difficulties that need to be solved to improve the accuracy of reconstruction missing markers. The increasing number of gaps or the duration of sequence causes the degradation of performance significantly [10]. Gloersen et al. [9] claimed that the reconstructions are affected by the number of missing markers simultaneously.
Moreover, the problem remains challenging when the given testing data comprise more than a single motion type that causes the motion to be less predictable (for example: a motion sequence includes both walking and running motion) In this thesis, we aim to deal with the mentioned difficulties, we exploit the principal component analysis (PCA) to recover the missing markers. We propose approaches based on the PCA concept that could make use of the current motion sequences. From the Mocap Dataset, we extracted a set of samples. Then, using PCA on the collection, a set of basis systems is generated, which eliminates the problem of lacking un-missing frames.
The weight vector for basis systems is then computed. Each sample in the collection contributes differently to the final interpolation up to the computed weight vector. As a result, the techniques are applicable to various types of markers and types of motion. Finally, interpolation is a synthesis of the basis systems with their weight.
Our methods robust the effectiveness of the original PCA method and handle the remaining problems. 4 Figure 3: Examples of a motion sequence with 1 missing marker. The missing marker has coordinate (0,0) is indicated in black circle. Contributions and Thesis’s layout The main contribution is approaches employ principal component eigenspace based learning mechanism is presented; The rest of my thesis is outlined hereafter.
Chapter 2 provides briefly several recent approaches. In Chapter 3, we state my methods. Chapter 4 comprises the design of experiments to clarify our algorithm’s performance. In the last Chapter, we present the conclusion and potential works in the future.
RELATED WORK From our perspective, the recent gap reconstruction approaches can be roughly categorized into groups as follows. Polynomial interpolation Many commercial marker-based motion tracking systems include polynomial interpolation as a typical gap filling approach. Examples of this type of approach include linear interpolation, spline interpolation, and monotone piecewise cubic interpolation [11]. They are usually suitable for tiny intervals, approximately 0.2 seconds [12], but not for large gaps.
As a result, various new interpolation algorithms have been proposed to take advantage of the available temporal or spatial information. Kalman filters, for example, can be used to estimate missing markers in a brief sequence [13]–[15]. In addition, real-time applications can benefit from approaches based on Gaussian process dynamic models [16] or linear dynamic systems (LDS) [17]. Human motion has also been modelled using the Hidden Markov Model (HMM) [18].
Furthermore, various techniques based on principal component analysis have been created (PCA). They do, however, lack a training sample set. Federolf [5], for example, used the mapping between PCA spaces to interpolate motion data. However, if there are gaps in numerous markers or the movements are less predictable or cyclic, this strategy is unsatisfactory.
This is due to the inability to adjust the number of principle component eigenvectors adaptively. Gloersen et al. [10] used a data-driven approach to reduce the overfitting problem in least squares by assigning weights to individual gaps and omitting some markers' trajectories. These attempts yielded promising but not perfect outcomes, especially in the case of motion data with less predictable or recurrent movement patterns.
6 Liu and McMillan [19] used a projection onto PCA eigenspace to fill in the missing data, then refined the estimates using a local linear model generated by the Random Forest classifier. However, while these methods are normally effective for small gaps (typically less than 0.5 second for human full-body motion), they can be insufficient and fail when used to bigger gaps [9], as discussed in section IV-B. While these models can be more or less effective, the lack of training data makes them less resilient. Figure 4: Block diagram of the Tits method [9], probabilistic averaging method of several recovery models 7 2.
Low-rank decomposition techniques Methods based on low-rank decomposition approaches have recently demonstrated promising results [20], [8], [25], [21].