HO CHI MINH CITY NATIONAL UNIVERSITY UNIVERSITY OF INFORMATION TECHNOLOGY DEPARTMENT OF COMPUTER SCIENCE Tran Kim Hung GRADUATION THESIS SPEAKER ADAPTATION IMPROVEMENT METHODS IN SPEECH SYNTHESIS BACHELOR OF COMPUTER SCIENCE HO CHI MINH CITY, 2021 HO CHI MINH CITY NATIONAL UNIVERSITY UNIVERSITY OF INFORMATION TECHNOLOGY DEPARTMENT OF COMPUTER SCIENCE Tran Kim Hung - 18520811 GRADUATION THESIS SPEAKER ADAPTATION IMPROVEMENT METHODS IN SPEECH SYNTHESIS BACHELOR OF COMPUTER SCIENCE THESIS ADVISORS: Trinh Quoc Son M.Sc Ngo Duc Thanh Ph.D HO CHI MINH CITY, 2021 THÔNG TIN HỘI ĐÒNG CHÁM KHÓA LUẬN TÓT NGHIỆP Hội đồng chấm khóa luận tốt nghiệp, thành lập theo Quyết định số 36 ngày 17/1/2022 của Hiệu trưởng Trường Đại học Công nghệ Thông tin. ACKNOWLEDGEMENTS I would like to express my sincere gratitude to my advisor and instructor Trinh Quoc Son. With the enthusiasm and support, he gave me, I embark on exploring the realm of speech synthesis and voice cloning domain in computer science. My sincere appreciation also goes to the co-supervisor of the thesis, Dr.
Ngo Duc Thanh from the department of computer science, University of Information Technology. He had taken his valuable time to review my work and give me direction on my thesis. If not for him, I couldn’t complete my research. My last gratitude goes to the council of the department of computer science had gave me a chance to work on this thesis.
TABLE OF CONTENTS ABSTRACT.ccscsscssssssssessesessesssessssesessesesnesesessenesnssesessesesessesessesesecsesecuesesecsesesuesesecaeseaeeseseenenes al CHAPTER 1. Scope and Goal: 1. History of TTS. Advantages of Neural Network Over Hand-crafted Methods 2.
Data Training Efficiency. Automation in Neural Speech Synthesis. Quality of Sound. Why Does Speaker Adaptation Important 2.
Existing Methods of Speaker Adaptation 2. Tuning Speaker Adaptation 2. Zero-Shot Speaker Adaptation CHAPTER 3. The Downsides of RNN.
Sequence-to-Sequence Architecture 3. Generalized End-to-End Loss 3. How Speaker Embedding Improves Speech Synthesis 3. Model Training CHAPTER 4.
SV2TTS Architecture Limitation. Proposed Methods of Improvement 4. Modification to The Flow of Speaker Embedding 4.3, Add More Information To The Speaker Encoder CHAPTER 5. REFERENCE 2PPPĐSSS.
54 LIST OF FIGURES FIGURE 1.1: WORKFLOW OF A SPEECH SYNTHESIS SYSTEM .1: FABER WONDERFUL TALKING MACHINE.2: MODULES OF A UNIT SELECTION TTS SYSTEM [2] .3: STATISTICAL PARAMETRIC SPEECH SYNTHESIS [2].4: THE PROCESS OF RETRAINING ANOTHER SPEAKER .5: THE PROCESS OF SPEAKER ADAPTATION TO A TUNING-BASED NEURAL SPEECH SYNTHESIS .6: ZERO-SHOT SPEAKER ADAPTATION PIPELINE [8].7: THE PROCESS OF SPEAKER ADAPTATION IN THE ZERO-SHOT SPEAKER ADAPTATION METHOD .1: EXAMPLE SYNTHESIS OF A SENTENCE IN DIFFERENT VOICES USING THE SYSTEM. MEL SPECTROGRAMS ARE VISUALIZED FOR REFERENCE UTTERANCE USED TO GENERATE SPEAKER EMBEDDING (LEFT), AND THE CORRESPONDING SYNTHESIZER OUTPUTS (RIGHT). THE TEXT-TO-SPECTROGRAM ALIGNMENT [8].2: RECURRENT NEURAL ÑETWORK.3: LONG SHORT TERM MEMORY.5: THE DIFFERENCES BETWEEN CONVENTIONAL RNN, LSTM, AND GRU .6: ENCODER-DECODER ARCHITECTURE [2].7: ALIGNMENT MECHANISM PROPOSED BY BAHDANAU ET AL [2].9: SIMILARITY MATRIX CONSTRUCTION AT TRAINING [7] .10: VISUALIZATION OF SPEAKER EMBEDDINGS EXTRACTED FROM LIBRISPEECH UTTERANCES. EACH COLOR CORRESPONDS TO A DIFFERENT SPEAKER.
REAL AND SYNTHETIC UTTERANCES APPEAR NEARBY WHEN THEY ARE FROM THE SAME SPEAKER, HOWEVER REAL AND SYNTHETIC UTTERANCES CONSISTENTLY FORM DISTINCT CLUSTERS.11: UMAP PROJECTIONS OF UTTERANCE EMBEDDINGS FROM RANDOMLY SELECTED BATCHES FROM THE TRAIN SET AT DIFFERENT ITERATIONS OF OUR MODEL. UTTERANCES FROM THE SAME SPEAKER ARE REPRESENTED BY A DOT OF THE SAME COLOR. WE SPECIFICALLY OMIT TO PASS LABELS TO UMAP, SO THE CLUSTERING IS ENTIRELY DONE BY THE MODEL.12: INPUT & OUTPUT OF SPEAKER ENCODER .13: INPUTS & OUTPUT OF SYNTHESIZER SYSTEM.14: TACOTRON 2 WITH SPEAKER EMBEDDING ARCHITECTURE [3' FIGURE 3.15: INPUT & OUTPUT OF VOCODER.16: FATCHORD WAVERNN ARCHITECTURE [3] .17: THREE STAGES TRAINING OF THE MODEL [3].1: DIFFERENT MODES OF THE MULTIMODAL SPEAKER ADAPTIVE ACOUSTIC ARCHITECTURE. DASHES BORDER INDICATES MODULES WITH TRAINABLE PARAMETERS WHILE BOLD SOLID BORDER INDICATES MODULES WITH IMMUTABLE PARAMETERS.2: ARCHITECTURE OF DEEP VOICE 3.3: MULTI-SPEAKER LDE TTS SYSTEM.
ENCODER BLOCKS ARE IN ORANGE, DECODER BLOCKS IN BLUE, POST-NET BLOCK IN GREEN, SPEAKER ENCODER BLOCK IN RED, AND VOCODER BLOCK IN YELLOW [10].4: OUR PROPOSED ARCHITECTURE TO PASS THE SPEAKER EMBEDDING TO THE POST-NET AND PRE-NET LAYER (WE HOWEVER DID NOT APPLY THE SPEAKER EMBEDDING TO THE OUTPUT OF THE POST-NET).5: LST-TTS SYSTEM [12] .6: WAV2VEC FEATURE EXTRACTOR OVERVIEW [13] Ad FIGURE 4.7: OUR PROPOSED MODEL BY CONCATENATING WAV2VEC2 SPEAKER REPRESENTATION AND GE2E SPEAKER REPRESENTATION. LIST OF TABLES TABLE l.1: METHODS OF SPEAKER ADAPTATION. ‘OMPARISON BETWEEN THE THREE SPEECH SYNTHESIS METHODS [21] TABLE 2.2:THE RESULT WHEN EXPERIMENTS WITH TACOTRON2 [6].3:RESULT OF TUNING MODEL [22].4: SPEAKER SIMILARITY MOS WITH 95% CONFIDENCE INTERVALS [8].1: OUR EXPERIMENT ON THE SV2TTS SYSTEM.2: CROSS-DATASET EVALUATION FOR UNSEEN SPEAKERS [8] TABLE 4.3: PERFORMANCE BETWEEN DIFFERENT TRAINING LOSS FUNCTIONS [7].4: PERFORMANCE USING SPEAKER ENCODERS (SES) ON DIFFERENT DATASETS [8].1: CRITERIA FOR DIFFERENT EVALUATION MEASUREMENTS. TABLE EXPERIMENT RESULT OF THE PROPOSED MODEL TRAINED ON DI TABLE 5.3: COMPARISON BETWEEN ARCHITECTURES TABLE 5.4: COST OF TRAINING AND TUNING.
ABBREVIATIONS LSTM Long Short Term Memory TTS Text-to-Speech RNN Recurrent Neural Network MOS Mean Opinion Score NMOS Naturalness Mean Opinion Score SMOS Similarity Mean Opinion Score BLSTM Bidirectional LSTM CBHG 1-D convolution bank + highway network + bidirectional GRU GRU Gated Recurrent Unit STFT Short Time Fourier Transform DNN Deep Neural Network SV2TTS Speaker Verification To Text To Speech GPU Graphic Processing Unit CPU Core Processing Unit ITU International Telecommunication Union EER EER: Equal Error Rate ABSTRACT Text-to-Speech (TTS) synthesis is an automatic conversion of written contents to spoken language. TTS synthesis plays a critical role in natural human and computer interaction in an organic manner. Although, communication between man and machine can be satisfied by commands and text appearing on the screen. Some applications such as Siri or Cortana are the prime example of what communication between man and machine can take shape with help of TTS.
Classic approaches in speech synthesis are limited to a definitive set of data and heavy loads of hand-crafted procedures (rule-based algorithms used to perform linguistic analysis). In this epoch of neural networks, improvements to the quality of sound and flexibility of training a TTS model using neural networks have been increased rapidly throughout the years of development. However, there are questions still open for improvement. One of many problems is the issue of creating an appropriate model for speech production with minimal resources and time.
Our objective is to tackle this topic and investigate what we can contribute to it. Many pieces of research have been conducted to address this concern; it’s known as the problem of speaker adaptation. There are various proposed methods to address the speaker adaptation problem, tune the model weight, and design a multi-speaker model were the conventional method of speaker adaptation. In this work, we want to investigate how they impacted the problem, propose solutions to improve the quality of the generated speech, and reduce the time needed to deliver the appropriate model for speech synthesis via tuning the weights and modifying the architecture.
The definition of speech synthesis is the process of artificially generating human speech from text. It aims to synthesize intelligible and natural audio indistinguishable from human recorded audio. A speech synthesizer is a computer system designed for this particular purpose. Speech synthesis has applications that can be useful for people with disabilities and dyslexia.
One of the most iconic application cases is Dr. The TTS system is the automatic converter of written to spoken language. The input is text, and its mission is to generate a speech waveform that corresponds to the original text (Figure 1. Speech Synthesis Input text seen Figure 1.1: Workflow of a speech synthesis system 1.
Problem Statement In speech synthesis, it is essential to produce a system that can generate sound identical to the speaker. It aims to replicate both in terms of naturalness, styles, and similarity in tempo. However, it is also crucial to train and produce the system model fast and efficiently to compete with other rivals in this growing market of artificial voice cloning. There are many challenges concerning this topic.
They have been gatekeeping the breakthrough in this field. Those problems are such as: - Fear of violation to the freedom of speech: With the improvement of speech synthesis naturalness throughout the years of development, it is natural for the mass to fictionalize a dystopia where their voices are taken advance of for malicious intents, and the more the field reaches its breakthrough, the more their fear becomes a reality. In response to the fear, regulations are set to prevent the collection of voice data and make it harder to train the model. This fear is proven to be a hindrance to the development of the field of speech synthesis.
To generate a system model that can achieve the naturalness of a desired human speaker, the researcher needs to have an extravagant amount of high-quality data. Collecting speeches data with high quality in large quantities can be problematic to compromise. With the fear of losing the freedom of speech as the base, retaliation from the people is inevitable. There also are laws that address this issue authorized to protect their voices from malicious intentions.
To solve this case, we not only need to strategize a proposal to model a system with very little data we have, but it also provides output with decent quality. - Similar to the problem of data-hungry, to generate a system that reaches the minimum requirement to a human-like product, massive computing power is needed. However, this work was composed in a global chip shortage crisis, which made it impossible to own a system capable of generating an appropriate system model for speech production in a short amount of time. Therefore, we need to find a solution to run the algorithm in low computing configuration.
Scope and Goals 1. Scope The scope of our work is to survey existing work related to the problem of saving time on producing a model for speech synthesis. During the endeavor to consult this subject, we discovered what is known as speaker adaptation. It's abridged as a set of techniques that allow researchers to generate speech for a specific speaker with very minimal data and computing power.
There are many related works on the study of speaker adaptation (Table 1. The details will be elaborated on further in section 2.5 of this thesis. Adaptation paradigm Methods LDE-TTS [10] Zero-Shot Speaker Adaptation Deepvoice 3 [11] SV2TTS [8] Tacotron [21, 22] Tuning-based Speaker Adaptation Tacotron 2 [2, 5, 6] Table 1.1: Methods of speaker adaptation 1. Goals The first goal of this work is a legacy continuation of [8], which is to build a TTS system that can generate natural speech for various speakers in a data- efficient manner.
We are based on an existing zero-shot learning setting, where several few-seconds of un-transcribed reference audios from a target speaker are used to synthesize new speech in that speaker’s formant, without updating any model parameters. Such systems have accessibility applications, such as restoring the ability to communicate naturally to users who have lost their voice and cannot provide many new training examples. They could also enable new applications, such as transferring a voice across languages for more natural speech-to-speech translation or generating realistic speech from texts in low- resource settings. However, it is also important to note the potential for misuse of this technology, for example, replicating someone’s voice without their consent.
The second goal of our work is to optimize the adaptation process and output speech quality so that it is undifferentiated from the original human voice. We also want to accelerate the production operation for an appropriate model for the speech generating system, which means we want the model training to be faster to save time. Contributions In this thesis, we survey all research related to the field of speaker adaptation. We aim to resolve which method would bring the most noticeable improvement.
We also combine some of the proposed methods. The justification for fusing them is to observe how they complement each other. By orchestrating them together, it allowed us to inspect what kind of impact they pose on one another.