VIETNAM NATIONAL UNIVERSITY, HANOI INTERNATIONAL SCHOOL GRADUATION PROJECT A Comparative Study of Variational Autoencoders with Different Encoder-Decoder Architectures for Time-Series Data Generation Tran Ngoc Thanh Binh Hanoi - 2024 VIETNAM NATIONAL UNIVERSITY, HANOI INTERNATIONAL SCHOOL GRADUATION PROJECT A Comparative Study of Variational Autoencoders with Different Encoder-Decoder Architectures for Time-Series Data Generation SUPERVISOR: Dr. Nguyen Quang Thuan STUDENT: Tran Ngoc Thanh Binh STUDENT ID: 20070902 COHORT: QH-2020-Q SUBJECT CODE: INS401101 MAJOR: Business Data Analytics Hanoi - 2024 A Comparative Study of Variational Autoencoders with Different Encoder-Decoder Architectures for Time-Series Data Generation Tran Ngoc Thanh Binh 3 Acknowledgement First, I want to send my sincere gratitude to my advisor, Dr. Nguyen Quang Thuan for the continuous support and guidance of my thesis, for his unwavering belief in me and this thesis, and for his immense knowledge. I want to thank all the members of the International School’s Club of Science and Technology’s Data Science team, for allowing me to work with them for the last three years, your enthusiasm and guidance are there to be remembered.
Finally, I would like to thank my family and friends who have always supported my journey in college. 4 Abstract The explosive growth of Large Language Models (LLMs) in the past year has raised significant interest in acquiring as much data as possible. The main problem is not all of the data can be acquired or should be acquired, this problem concerns the privacy, and copyrights of Internet users, copyright holders, and constitutions around the world. Hence, synthetic data has been gaining traction as a powerful solution to the challenge of privacy and diversity of data.
This project focuses on a subset, which is Time Series data generation, the need for Time Series Synthetic Data is highly concerned in various applications, including data generation, and privacy preservation. This comparative study concerns the efficacy and accuracy of Time Series data generation methods, particularly those based on Variational Autoencoder. Autoencoders are artificial neural network architectures that are intended for the compression and reconstruction of data. Variational Autoencoder (VAE), in particular, achieves so by introducing a probabilistic view of encoding and using stochastic variational inference.
VAE has been widely used for data reconstruction or generation. This thesis aims to test different variational autoencoders generating Time Series data. This thesis investigates two novel implementations of VAE, to find out the strengths and weaknesses of each architecture on different types of Time Series data. I compare the generative capability of three VAE-based architectures on various types of Time Series data.
This project uses an established framework that includes a standardized prepro- cessing pipeline and systematic evaluations.1 Time Series data in different domains .2 Importance of synthetic data .3 Ethical Consideration for Synthetic Data Generation .4 Types of synthetic data generators .5 Existing generative methods .6 Chosen VAE-based methods for comparison .7 Contribution of this Thesis. 13 2 Related Work and Background 14 2.2 Synthetic data generation .3 Artificial Neural Networks .1 Recurrent Neural Networks .2 Convolutional Neural Network .1 Short-Time Fourier Transform .2 Kullback-Leibler Divergence .3 Evidence Lower Bound (ELBO). 26 3 VAE-based Time Series Generation methods 26 3.1 Stage 1: Learning Vector Quantization .2 Stage 2: Prior Learning .1 Base TimeVAE Architecture. 30 4 Method for Comparison 31 4.2 Final methods for comparison .1 Metrics for TSGBench comparison .1 TSG Benchmarking Results .1 TimeVAE’s Ablation Study .2 TimeVQVAE’s Ablation Study.
43 6 6 Conclusions and Future Work 44 7 List of Figures 2.1 Structure of a feed-forward neural network .2 An RNN with hidden state.3 Two-dimensional cross-correlation operation. The output is calculated as 0 × 0 + 1 × 1 + 3 × 2 + 4 × 3 = 19 .4 Given a handwritten digit image, LeNet performs a series of computations to classify it into one of 10 categories, yielding a probability for each category as its output.5 The attention mechanism calculates a weighted combination of values v via at- tention pooling, these weights are determined by how well each query q matches or aligns with the corresponding keys ki .6 Multi-head attention.7 The Transformer architecture.8 Structure of an Autoencoder, extracted from [Wikipedia contributors, 2024] .9 Graphical model representation of VAE. Given N observed data points {xi }, each data point is locally generated by a latent random variable zi. θ is a global parameter, and is obtained through training .1 Overview of Stage 1 - Learning Vector Quantization .2 Stage 2 - Prior Model Training.
Dark green blocks represent the masked tokens.3 Iterative decoding process with two passes. TF-LF and TF-HF denote the LF and HF bi-directional transformers respectively.4 Components of base TimeVAE .5 Interpretable TimeVAE components, extracted from [Desai et al.6 Trend and Seasonality Blocks, extracted from [Desai et al.1 TSG Benchmarking result.2 TSG Bench Visualization by t-SNE and Distribution plot, blue as T org and orange as T gen .3 Dense VAE vs Time VAE on Original vs Reconstructed Train for Air dataset.4 Dense VAE vs Time VAE on Original vs Reconstructed Train for Energy dataset.5 Dense VAE vs Time VAE on Original vs Reconstructed Train for Sine dataset.6 Dense VAE vs Time VAE on Original vs Reconstructed Train for Stockv dataset.7 Dense VAE vs Time VAE t-SNE plots Train for Air dataset.8 Dense VAE vs Time VAE t-SNE plots Train for Energy dataset.9 Dense VAE vs Time VAE t-SNE plots Train for Sine dataset.10 Dense VAE vs Time VAE t-SNE plots Train for Stockv dataset.11 Convolutional VAE vs Time VAE on Original vs Reconstructed Train for Air dataset.12 Convolutional VAE vs Time VAE on Original vs Reconstructed Train for Energy dataset.13 Dense VAE vs Time VAE on Original vs Reconstructed Train for Sine dataset.14 Dense VAE vs Time VAE on Original vs Reconstructed Train for Stockv dataset.15 Dense VAE vs Time VAE t-SNE plots Train for Air dataset.16 Convolutional VAE vs Time VAE t-SNE plots Train for Energy dataset.17 Convolutional VAE vs Time VAE t-SNE plots Train for Sine dataset.18 Convolutional VAE vs Time VAE t-SNE plots Train for Stockv dataset.19 FID and IS score for VQ-VAE and TimeVQVAE .20 VQ-VAE vs TimeVQVAE on reconstructing examples and generated samples.1 Turnitin Similarity Score of this Thesis. Time Series Generation ANN. Artificial Neural Network VAE.
Variational Autoencoder CNN. Convolutional Neural Network LLM. Large Language Model STFT. Short-Time Fourier Transform ISTFT.
Inverse Short-Time Fourier Transform KL. Kullback-Leibler ELBO. Evidence Lower Bound VQ. Vector Quantization LF.
Low-Frequency HF. High-Frequency Conv. Fréchet Inception Distance C-FID. Contextual Fréchet Inception Distance ED.
Euclidean Distance DTW. Dynamic Time Warping 9 1 Introduction In the past decades, with the boom of the internet, coupled with humanity’s endless effort in advancing science and technology, we have seen a rapid growth of Machine Learning and Deep Learning methods to solve problems that we have not been able to solve before. Fields including Computer Vision (CV), natural language processing (NLP),. Problems involving time series have also been increasingly attempted and tacked using deep learning, including problems in classification [Ismail Fawaz et al., 2019], forecasting [Han et al., 2019], and anomaly detection.
The success of applying deep learning to those problems requires having a large amount of data. Unfortunately, time series tasks typically do not have enough data for such models. As we try to resolve this problem, data generation has become an effective tool to increase the size and quality of data. The main idea of applying data generation is to try to synthesize data points that are realistic.
The recent boom of Large Language Models, further signifies the need for time series data as we are trying to improve LLMs’ time series analysis capabilities [Zhang et al. There are many methods for time series generation, including basic methods based on Time Domain and Frequency Domain to more advanced methods like Statistical Generative Models. However, Deep Generative Models remain less investigated for time series data generation. In this Thesis, we explore the usage and results of two such models, both based on Variational Autoencoders [Kingma and Welling, 2013].1 Time Series data in different domains Time series data is widely used in different fields and has become crucial in predicting and forecasting potential needs.
Time series data can be utilized in predicting the risk of disease and providing people with medical help. Time series can also be used by governments and companies to make decisions on energy, climate, and finance. The growth is exponential, as the amount of massive data encourages people to explore various applications. However, due to problems in quality, quantity, and privacy of using real data, people usually do not use the original data when exploring applications in various domains.
I give examples within several representative domains to show how these problems with real data drive the necessity for generated data.1 Healthcare Time series data is used in healthcare to make more accurate diagnoses. • Time series data is used to effectively predict the blood glucose level of patients, which is critical for diabetes subjects [Bhimireddy et al. • Wearable devices also collect large amounts of data and can provide suggestions to improve people’s health. For example, [Sathyanarayana et al., 2016] uses wearable devices for sleep condition tracking.
Data collected by wearable devices can include location, sound, and images, all of which are very sensitive and should only be stored on-device. Algorithms and methods [Bonawitz et al., 2017] [Jayaraman et al., 2018] have been developed to train machine learning models on these kinds of data without sending them to a centralized server. Healthcare machine learning systems analyze highly sensitive data and impact critical decisions. Ensuring data quantity, quality, balance, and privacy is crucial.2 Energy Time series data is also relevant in the energy sector, having been used to predict and monitor the power usage of each home appliance or the whole grid.
• [D’Incecco et al., 2019] used the UK-DALE dataset [Kelly and Knottenbelt, 2015], which includes records for each appliance from five houses, one of which was recorded for 655 days to perform Non-Intrusive Load Monitoring through Transfer Learning schemes. • Time series data in this field can also be used to forecast the load of an entire district’s heating system. [Gong et al., 2022] uses data from a District Heating System in Tianjin, China to perform the task. Data in energy systems is hard to record.
Especially household appliances’ energy consumption data due to privacy concerns, cost, and the quality of data.2 Importance of synthetic data Despite the huge amount of money invested every year by institutions to collect time series data, real time-series data can not always satisfy all the needed characteristics. Overall, there are time-series data-related issues in various domains: • Quantity issue: In certain areas, the amount of data we can get is insufficient. Especially if the acquisition of such data requires people with specialized skills, for example in the healthcare domain, the amount of data we can get and the cost to get it will be a problem. If we can utilize synthetic data to supplement and enhance the existing actual data, then more applications can be built without requiring more real data than we currently possess.
• Quality issue: Quality issues with data are common. During the acquisition process, there are various factors that can cause inconsistency in the quality of data. For instance, a questionnaire can have missing values and outliers simply because people incorrectly filled the questionnaires. • Imbalance issue: Data imbalance in time series data is normal.
This is due to the nature of real-world phenomena, where certain events or patterns naturally occur less frequently than others. Imbalance poses challenging problems when developing models. This problem can be mitigated by using synthetic data to supplement the niche parts of the dataset. • Privacy issue: Privacy is always a problem when it comes to data acquisition.
Data that contain sensitive information is usually strictly protected and researchers often can not get access to those data.