NGUYỄN TUYÊN HOÀNG MATÉRN KERNEL FOR CONTINUOUS SMOOTHNESS PARAMETER IN GAUSSIAN PROCESS BASED ALGORITHMS LUẬN VĂN THẠC SĨ NGÀNH ĐIỆN TỬ, NĂNG LƯỢNG ĐIỆN, TỰ ĐỘNG HÓA CHUYÊN NGÀNH KỸ THUẬT TRUYỀN THÔNG VÀ DỮ LIỆU NGƯỜI HƯỚNG DẪN KHOA HỌC GS. EMMANUEL VAZQUEZ TS. TRẦN QUỐC LONG HÀ NỘI, NĂM 2024 MATÉRN KERNEL FOR CONTINUOUS SMOOTHNESS PARAMETER IN GAUSSIAN PROCESS BASED ALGORITHMS Master thesis of Paris-Saclay University and VNU University of Engineering and Technology Specialization: M2 Data and Communication Engineering Research unit: Advanced Institute of Engineering and Technology, VNU University of Engineering and Technology Thesis presented at Hanoi, on 30 January 2024 NGUYEN Tuyen Hoang Committee Arnaud BOURNEL Paris-Saclay University Chairman Pierre DUHAMEL CNRS, CentraleSupelec, Paris-Saclay University Rapporteur NGUYEN Linh Trung VNU University of Engineering and Technology Examiner Emmanuel VAZQUEZ CNRS, CentraleSupelec, Paris-Saclay University Examiner TRAN Quoc Long VNU University of Engineering and Technology Examiner Master Thesis Thesis Supervision Emmanuel VAZQUEZ CNRS, CentraleSupelec, Paris-Saclay University Supervisor. TRAN Quoc Long VNU University of Engineering and Technology Co-supervisor.
First, please allow me to thank Prof. Emanuel Vazquez and PhD. Tran Quoc Long, your encouragement and guidance always give me more motivation in the process of researching and implementing this thesis. I would like to send my most sincere thanks to Paris Saclay University and University of Engineering and Technoly – Vietnam National Universit for nurturing and teaching me useful knowledge during over the past year.
I would also like to thank the teachers and students in the M2 – Communication and Data Engineering Course for their advice, suggestions and support throughout my time studying at school to help me complete my thesis in the best possible way. Last but not least, I would like to thank my parents, my family and my friends for always being by my side and encouraging me to complete this study program. Authorship I solemnly declare that my thesis, titled ’MATÉRN KERNEL FOR CONTINUOUS SMOOTHNESS PARAMETER IN GAUS- SIAN PROCESS BASED ALGORITHMS’, is my own research work conducted under the guidance of Prof. Emmanuel VAZQUEZ and Prof.
TRAN Quoc Long. The sources used in the thesis are explicitly mentioned in the reference sec- tion, with proper citations. The data and results presented in the thesis are entirely truthful, and there is no copying from the works of others. If any discrepancies are found, I take full responsibility and am subject to any disciplinary actions imposed by the university., 2024 Student Nguyen Tuyen Hoang Abstract Gaussian Process (GP) regression has gained significant attention due to its ability to model complex and non-linear relationships in data.
The Matérn kernel is a popular choice in GP-based algorithms, offering flexibility in captur- ing various levels of smoothness in the underlying function. However, the traditional approach of discretizing the smoothness parameter limits its expressive power and introduces computational challenges. This thesis focuses on addressing these limitations by introducing a novel framework that enables the use of a continuous smoothness parameter within Matérn kernels. We propose a parametric approach that allows the smoothness parameter to take on any real value, rather than being restricted to a discrete set.
This continuous smoothness parameter provides greater flexibility in modeling diverse data patterns, particularly those that do not conform to the predefined discretized levels of smoothness. To achieve this, we develop a generalization of the Matérn kernel that incorporates the continuous smoothness parameter. We derive the necessary mathematical formulations and establish its properties and behavior. We also propose a scalable algorithm for estimating the continuous smoothness parameter from data, thereby enabling automatic adaptation to the underlying function’s complexity.
We conduct extensive experiments on synthetic datasets as well as real-world applications, such as time series prediction and spatial modeling. The results demonstrate the effectiveness of the proposed framework in captur- ing a wide range of smoothness patterns, outperforming existing methods using discrete smoothness parameters. Moreover, the computational efficiency of our approach allows for practical implementation in large-scale datasets. The contributions of this thesis extend beyond the development of a continuous smoothness parameter for Matérn kernels.
We also explore the theoretical implications of using continuous parameters in GP-based algorithms and provide insights into the impact on model interpretability and hyperparameter optimization. In conclusion, this thesis presents a novel framework that enables the use of a continuous smoothness param- eter in Matérn kernels within Gaussian Process-based algorithms. The proposed approach offers greater flexibility, improved modeling capabilities, and enhanced computational efficiency. The experimental results validate its effec- tiveness in various applications, showcasing its potential for advancing the field of Gaussian Process regression and its practical applications in diverse domains.
Contents List of Figures 4 1 Introduction 5 1. 7 2 Background and Related Works 8 2.1 Why is Gaussian Process? .4 Modeling and Predicting for Gaussian Process .1 Gaussian Process Regression (GPR) .1 Maximum Likelihood Estimation .5 Sequential Model Updating .2 Prediction of a zero-mean GP .3 Prediction of a zero-mean GP with noise .1 Bessel function of the first kind Jν (z) .2 Bessel function of the second kind Yν (z) .3 Modified Bessel Function .1 Modified Bessel function of the first kind .2 Modified Bessel function of the second kind .4 Evaluate Modified Bessel Function .1 Power series expansion .1 BesselK accuracy evaluation experiment .2 Combine with logbesselk .3 Compare with fixed smoothness parameter. 33 5 Conclusions and Perspectives 34 Bibliography 36 3 List of Figures 2.1 Kriging prediction based on noisy observation .1 Predicted - Observed graph for besselk .2 Predicted - Observed graph for combined besselk .3 First dataset description .4 First dataset description .6 Model Diagnosis for Dataset 1D .7 Predicted - Observed graph for 2D Dataset .8 Model Diagnosis for Dataset 2D .9 Model Diagnosis with fixed smoothness parameter. 33 4 Chapter 1 Introduction Gaussian processes (GPs) have emerged as powerful tools for modeling and analyzing various types of data, offering flexible and non-parametric approaches to capture complex patterns and uncertainties.
A key component of Gaus- sian processes is the choice of a covariance function, also referred to as a kernel, which encodes the assumptions about the underlying smoothness and correlation structure of the data. Among the diverse set of kernels available, the Matérn kernel has gained significant attention due to its ability to represent a wide range of smoothness levels in the generated sample paths. The Matérn kernel is characterized by a smoothness parameter, typically represented as ν , which governs the differentiability of the sample paths. Unlike other kernels with fixed smoothness properties, the Matérn kernel allows for a continuous range of smoothness parameters, offering a fine-grained control over the modeling assumptions.
The choice of the smoothness parameter in the Matérn kernel plays a crucial role in capturing the desired char- acteristics of the underlying data. By selecting an appropriate value for ν , one can tailor the GP model to reflect the inherent smoothness or roughness in the observed data. For instance, a higher ν value results in smoother sample paths, which is suitable for data with high levels of differentiability or when modeling processes with con- tinuous and regular variations. On the other hand, lower ν values allow for more rough and irregular sample paths, accommodating data with abrupt changes or sporadic patterns.
Furthermore, the continuous smoothness parameter in the Matérn kernel provides an advantage in terms of model flexibility and interpolation capabilities. The ability to adjust ν allows for the capture of intricate and complex patterns, making the Matérn kernel well-suited for modeling spatial or temporal data exhibiting non-linear and irreg- ular variations. This flexibility enables Gaussian process-based algorithms to adapt to diverse datasets with varying degrees of smoothness, providing accurate interpolation and robust predictions. In this thesis, we aim to explore and investigate the utilization of the Matérn kernel with continuous smoothness parameter in Gaussian process-based algorithms.
We will investigate the impact of different ν values on the model- ing performance and predictive capabilities of Gaussian processes. Additionally, we will examine the computational 5 aspects associated with higher smoothness parameters and propose efficient algorithms to handle the increased complexity. By leveraging the flexibility of the Matérn kernel and its continuous smoothness parameter, this research aims to enhance the applicability and effectiveness of Gaussian process-based algorithms in various domains. The insights gained from this study can contribute to improved modeling, interpolation, and prediction of complex data patterns, enabling more accurate decision-making in areas such as spatial analysis, time series forecasting, and machine learning.
Overall, this thesis aims to shed light on the role of the Matérn kernel with continuous smoothness parameter in Gaussian process-based algorithms and provide valuable insights into its practical implications and potential for advancements in data-driven modeling and analysis.1 Problem Statement The Matérn kernel [Stein(1999)] is a popular covariance function used in GPs due to its flexibility and ability to tune the degree of differentiability of the GP with the smoothness parameter n > 0. When n is a half-integer, there exist convenient formulas for the kernel that allow for efficient computations. For example, when n = 1/2 , the kernel reduces to the exponential kernel. These convenient formulas make the Matérn kernel a popular choice in Gaussian process regression, particularly in situations where efficient computations are essential.
However, if one needs to optimize the value of n using a maximum likelihood approach, it is preferable to deal with a continuous value for n, which implies writing the Matérn covariance using the modified Bessel function of the second kind (Kν ). Unfor- tunately, there is no implementation of this function in pure Python, making it challenging to use this continuous version of the Matérn kernel in practice. So the objective is to write a pure Python implementation for the Kν and then apply it to optimize the smoothness parameter of the Matérn covariance using gradient-based optimization algorithms 1.2 Contributions To summarize, there are three main contributions of this thesis • Thesis give a general view about Gaussian Process smoothness parameter for Matérn Kernel and propose an appropriate way to integrate continuous smoothness parameter for Matérn Kernel in to GPmp. • Due to the absence of a pure Python code for Kν , the thesis endeavors to develop a dedicated Python code and integrate it into GPmp.
6 • The thesis conducts experiments and presents a comparison between a fixed smoothness parameter and a continuous smoothness parameter.3 Outline • Chapter 2 describes the basic knowledge of the Gaussian Process and some knowledge related to modeling and prediction for problems using the Gaussian Process. • Chapter 3 delves into the Bessel function, exploring related concepts and outlining the proposed technique for computing Kν. • Chapter 4 will describe the experiments that we conducted to evaluate the performance of the proposed method. • Chapter 5 will draw conclusions and future directions of the thesis.
7 Chapter 2 Background and Related Works 2.1 Why is Gaussian Process? Gaussian processes are a powerful probabilistic modeling technique that can capture complex patterns and un- certainty in data. Gaussian processes (GPs) have gained significant popularity in various fields, including machine learning, spatial statistics, and time series analysis, due to their flexibility, non-parametric nature, and ability to model complex data patterns. At the core of Gaussian processes lies the choice of a covariance function, or kernel, which characterizes the assumptions about the correlation structure and smoothness of the underlying data. In the age of deep learning, there are several reasons to use GPs, such as: • Flexibility and Interpretability: GPs provide a flexible framework for modeling complex relationships in data.
They can accommodate various forms of prior knowledge and can capture non-linear and non-parametric patterns. GPs also provide interpretable uncertainty estimates, which can be crucial in applications where understanding model uncertainty is important. • Small Data Regime: GPs perform well in scenarios with limited data. Unlike deep learning models that often require large amounts of labeled data to generalize effectively, GPs can make accurate predictions with small datasets.
GPs are particularly useful when data is scarce or expensive to collect.