Thesis for the Degree of Doctor Speech Emotion Recognition using Fuzzy Inference System based on Fuzzy Associative Memory 3] 2] ee2] ] o] 7) WAFEAADS AB r ^w|3| 4⁄4 a4) June 2014 Department of Digital Media Graduate School of Soongsil University TON THAT HOA AN Thesis for the Degree of Doctor Speech Emotion Recognition using Fuzzy Inference System based on Fuzzy Associative Memory June 2014 Department of Digital Media Graduate School of Soongsil University TON THAT HOA AN Thesis for the Degree of Doctor Speech Emotion Recognition using Fuzzy Inference System based on Fuzzy Associative Memory A thesis supervisor: Professor Hyung-I] Choi Thesis submitted in partial fulfillment of the requirements for the Degree of Doctor June 2014 Department of Digital Media Graduate School of Soongsil University TON THAT HOA AN To approve the submitted thesis for the Degree of Doctor by Ton That Hoa An Thesis Committee Chair (signature) Member (signature) Member (signature) Member (signature) (signature) June 2014 Graduate School of Soongsil University ACKNOWLEDGEMENT I would like to express my deep gratefulness to everyone who contributed to make my Ph. I am profoundly thankful to Soongsil University and Computer Vision lab which supported my tuition and subsistence fees during the years in Korea. I would like to express my especial thanks to my advisor, Professor Hyung-II Choi for his advices and help. My sincere gratitude goes to Professors in Digital Media department of SSU for their courses that I participated about Computer Vision, Image Processing, Pattern Recognition, Computer Graphics, Media Arts and other related topics.
I also greatly appreciate the help of Professor Michael T. Johnson (Marquette University, USA) whomI have never been met. I would like to gratefully acknowledge to my parents, my wife and my daughter for their endless support, encouragement and motivation. Also, I want to thank my friends in same lab as well as other labs for their good relation and cooperation, and Korean friends for helping me during my stays here.
Finally great thanks to all my colleagues and relatives for their support, and encouragement for the duration of my graduate research and especially to my committee for all of the insightful comments and encouragement along the way for finishing this dissertation. TABLE OF CONTENTS ABSTRACT IN ENGLISH----------- ---- 5222 S1} 9999 99222922221 111k kho ix ABSTRACT IN KOREAN rreerrsssrrrstert ttt nnn eens e eens XI CHAPTER 1 INTRODUCTION --:---::----------------------:---5©‡‡cŸSŸ‡‡<s-: 1 my mẻ.1 Rationale and Incentives _+eee°-~------ - - --- --- 2: -- nàn sà.2 Related Concepts and Definitions -----:---:-------------+--*-*+*reeseeeeeeeeeee 3 1.1 Applications of Speech Emotion Recognition ------:-+-++**+**+*+c*teeceecetec 4 1. «<M A ME Be cece cneeec ene rer ean eesens 5 1.3 Problem Statement -:--:-::::--:-:--:----+***** nh nh nh th nh hen nh kh th kg 8 1.4 nhicon in số.5 Dissertation Outline ---::--:-:-::----:-:++*+*rrrhhehehhhhhhthhhherhnnneeneeereeeeeeeee 9 CHAPTER 2 RELATED WORKS:----:---------------------------------------- 11 2.1 Speech Emotion Features --:--:::-::-::::::::+:::++trrttthhhhhhhhhhhhhhenehdee 11 2.1 Excitation Source Features ---:------::-::+:+++*rrerereereeeerrerereeeeeeeeeree 11 2.2 Vocal Tract Features <cccctctec cette eccrine th nh nh eee he he 12 2.3 Prosodic Features 100: :tttt tee etter nee eee: 13 2.4 Combination of Features -:---::--:-::-::+:+*rrrreenennrhentrnernerrreeeereeeerere 15 22 Emotion Classification Methods ¬.- 17 CHAPTER 3 FUZZY INFERENCE SYSTEM BASED ON FUZZY ASSOCIATIVE MEMORY -------------- 22 3.1 The Principal Structure of the Fuzzy Associative Memory --:-:------------- 22 3.2 Model of the FAM-based Fuzzy Inference System ---------:------::-:--:-:-- 23 3.3 Layers of the FAM-based Fuzzy Inference System ------------------------+-+ 24 3.4 Building up Membership Functions ---::::---------------*-*+*+‡cceeecececececằc 29 3.5 Basic Model of Inference and Fuzzy Rules ------:----:----:--:-+-++erreccererec 30 3.6 Determining Weight Matrix -------------------*****ee nhe hnhnhehehhheeeeeeeeeere 31 3.7 The Model of Speech Emotion Recognition using FAM-based Fuzzy Inference System Penne cere cence cree cere eres l9 9 9 sree eens eee sees ee eee eee neeeeneenenes 33 CHAPTER 4 SPEECH EMOTION RECOGNITION TOOLBOX -----:--------------------------------------Ÿcc cà: 35 4.1 Main Interface Window ----:::--:+** 22h nh nh hen 36 4.2 Features Group ---:::-:-:--++r+rerrrhthhhhhhhhnhhhh nen nh nh he khen nhe he 38 4.3 Framing Modes Group --::-::::::::::::++:trttthhhhhhhhhhhhhhhrrrnrrrerrre 39 4.4 Tools Group SS 42 4.1 Check Database Tool ---------:*+rrrehnnhnnhnnnnnnnenrnnnnneneenenneneereeeeree 42 4.2 Convert HTK to TXT Tool -::-:::::****22S $2 22** xxx theo 43 4.3 Cut Wav Files Tool ----:-:--------* + nh nh hề nh nh hề hề khe eens 44 4.4 Trim Wav Files Tool -:::---:::--:** 2$ 2h nhe reo 45 4.5 Resample Wav Files «111-1-:c eect etter tte ties 47 4.5 Operations Group --:::::+++++++rrrhhhhhhhhhhhhhhththhrrrrrrrrnnrnrrnnnnnnnrrtrrrreeee 48 4.6 Commands Group crisiseees 49 4.7 Recognition Group -----:-'::*******tthhhhhhhhhhhhhhthhhhenhenenheneeeereeeeeeereeee 50 4.1 CHE SVM Tool cecccecec tte nents nents 51 4.2 FAM FIS Tool ---:--------------== nh renee nee 52 CHAPTER 5 PERFORMANCE AND RESULTS ---------------------- 53 5.1 Databases «+++ ++ fae «+ aM eM Bice ese scene ens 53 5.1 Emo-DB Dafabase (Nn | A> -- ag, 1.2 SAVEE Database -----:-------------=* nh nh nền kh hề nền kh kh nhe nhe eh 56 5.3 Audio File Format <crccccctecetececee etter ne kh kh nh kh he he he 58 5.4 Database Processing ------::::::++++++rrrrthhhhhhhhhhhrrnnnnnnnnnrnnnnrnnrrre 60 5.2 Speech Emotion Feature Extraction -------::::::::+:+*rsreheneenenrnereeneeeerees 62 5.1 Mel Frequency Cepstral Coefficients (MFCCs) --------------------+*‡+Ÿccẻ 63 5.2 Generalized Frequency Cepstral Coefficients (GFCCs) -------:-------:-- 69 5.3 Generalized Perceptual Linear Prediction --------------------+++©*e+ecccccet 71 5.1 Pre-Emphasis Filter --------------*ssnnnn nh hhhhhhhh nh nh nh kh kh nh he he hen T2 5.2 Hamming Window «srr rere 3 5.3 Power Spectrum Estimation -:--:::--:::::::::+:::+r+rrrrrhhhhhhhhehrreh 74 523.4 Filter Bank Analysis ¬ 75 5.5 Equal Loudness Normalization -:-::::::::++:++++++r++rreereeeeeeceeececec 76 5.6 Intensity-loudness Power Law -::::+:*rrrrrrhhhhhhhrrrnrrnnnnrnrnnrnrrsree 16 5.7 Autoregressive Modeling -----::--:::::::::::+++:++tthhhnhhhhhhrenreo 6 5.8 Cepstral Domain Transformation -----:-+*+++++++**+eehetheeeeteeeeteeeerec 77 5.3 The Classification| RE 78 T1: .1 Emo-DB Database ---:::::--:--::::----+++* +22 St xe 80 5.2 SAVEE Database 00 eI 600 00000 SE en 82 5. 83 5422 Comparison Results Po Po SER Â.
(co e eo 86 CHAPTER 6 CONCLUSION AND FUTURE WORK --------------- 89 6.1 Overall Performance ------:-:--:--:-:++++++*+rreeeeeeeeeeeeeereeeereeeeeeeeeeeeere 89 6.2 Suggestion of Applications --:::::::::::+:++:t+tttttrthhrhhhhhhhhereesree 90 6.3 Future Work ccc ee eee EEEeee e 91 REFERENCES " 92 APPENDICES 0100200000022. 102 -iv- LIST OF TABLES [Table 5-1] Code and information of speakers in Emo-DB database------------------ 54 [Table 5-2] Code of emotions of Emo-DB database---------------------------+-ccccccccc: 54 [Table 5-3] Number of samples of emotions of Emo-DB database -------------------- 55 [Table 5-4] Notations of emotions used in Emo-DB database -:------:------------- 55 [Table 5-5] Number of samples of emotions of SAVEE database ---------------:----- 57 [Table 5-6] Notations of emotions used in SAVEE database ----------------------- 58 [Table 5-7] Recognition results of FAM FIS method on Emo-DB database in comparison with Support Vector Machine method ----------------------- 80 [Table 5-8] Confusion matrix of 19-dimension MFCC feature of Emo-DB D atabase đ :-——. 81 [Table 5-9] Recognition results of FAM FIS method on SAVEE database in comparison with Support Vector Machine method -----:--:--:--::---- 83 [Table 5-10] Confusion matrix of 19-dimensions MFCC feature of SAVEE database ". e eee eee eee eee eee 85 [Table 5-11] Confusion matrix of 17-dimensions GFCC feature of SAVEE LIST OF FIGURES [Figure 2-1] Types of classifiers used for speech emotion recognition ----:--:-:-:-- 21 [Figure 3-1] The principal structure of the fuzzy associative memory -:-:--:---:-- 22 [Figure 3-2] Model of fuzzy inference system based on fuzzy associative Memory viet te eee eee 24 [Figure 3-3] Type 1 of membership functions --------:--------++++-+*‡‡ te ceecceọc 25 [Figure 3-4] Type 2 of membership functIOns-----------+*++******+‡en set e te èc 26 [Figure 3-5] Type 3 of membership functiOnS---------++++-+++++++*+‡*c*e*c se cececcìc 26 [Figure 3-6] Membership functions of output variable -:-:-:------------+-+-+-+-+++++ 27 [Figure 3-7] Histogram of a feature (utter 12) of Emo-DB:---------------:-+-+c-cc: 30 [Figure 3-8] Smoothed histogram of a feature (utter 12) of Emo-DB -------------- 30 [Figure 3-9] Basic model of inference using fuzzy associative memory ---:::-:--- 31 [Figure 3-10] Fuzzy rules-+:+0:0:s0s0ee etter terre 31 [Figure 3-11] The model of SER using FAM-based fuzzy inference system ------ 34 [Figure 4-1] Main interface window of SERT vivre 36 [Figure 4-2] Information on Global Window Size and Global Step Size ---------- 37 [Figure 4-3] The image of an audio file in SAVEE database ------------------------- 37 [Figure 4-4] Command buttons used for speech emotion reCOØTIfION---:---------- 37 [Figure 4-5] “Features” group <r: ieee eer eeie 38 [Figure 4-6] MFCC parameter configuration interface ---:::------::---::-::---:+--+- 38 [Figure 4-7] “Framing Modes” group_----::--::::::::::::++:*+:*+tttthhhhhhhhhhthtee 39 -vi- [Figure 4-8] The HTK mode or traditional framing mode --::---::-------:---------- 40 [Figure 4-9] The Fixed Step Size framing mode ---:::-:-:-:--------:-+-+-+-*ccccc cà 41 [Figure 4-10] “Tools” group crite ttt eerie 42 [Figure 4-11] Interface of Tool for checking database --::--------------------------- 43 [Figure 4-12] Interface of Tool for converting HTK files to TXT files ------------ 44 [Figure 4-13] Interface of “Cut wav files” tool -----------+****sehnhehhhhhhheneeeeeeere 45 [Figure 4-14] Interface of “Trim wav files” tool ---------::*****crhehehhheheneeeeeere 46 [Figure 4-15] Interface of “Resample wav files” tool ------:--::--++*ree chen 47 [Figure 4-16] Functions of “Operations” øf0up_--::-:*:*:*:*:+*+*******eeeeeeheeeeeeec 48 [Figure 4-17] Functions of “Commands” group ----:::::-:-:---*:*+*+*+*+**e*+*+*ese*ẻ 49 [Figure 4-18] The interface of “Batch Process” tool -:-:--:-:++++++*rreteeheeeeteeeeet 50 [Figure 4-19] Command buttons in “Recognition” group cists 51 [Figure 4-20] Tool for Support Vector Machine classification -:-:-:--------------- 51 [Figure 4-21] Tool for FAM FIS classification -------:------:+*++*+*rreeeeheeeeeeeeeet 52 [Figure 5-1] A sample of emotion speech in Berlin database (Emo-DB) --------- 56 [Figure 5-2] A sample of emotion speech in Surrey Audio-Visual Expressed Emotion database (SAVEE) aT 57 [Figure 5-3] The algorithm for trimming silence segments at beginning and the ending of wav files ---:--::--::::::::++:++*2ttthhhhhhhhhhhhhhhhhrreree 61 [Figure 5-4] MFCC feature extraction block diagram ----------:--++-+++*+*+c+reecìc 64 [Figure 5-5] An example of MFCC filter bank---------------------+-*+**‡c‡‡ sec.
67 [Figure 5-6] GFCC feature extraction block diagram ---:-----:---++*+*+*+*+*+c*ccccc+ 70 -Vvil- [Figure 5-7] GPLP block diagram -----:--::-::--::-::::::::+:+:++:++tttttrhtererrr [Figure 5-8] Chart of FIS vs. SVM classification for MFCC feature of Berlin Emo-DB database ----:-:-:-:--------:*s nh nh nh nh thề eens the he kh th ens [Figure 5-9] Chart of SVM classification for SAVEE database ---------------------- [Figure 5-10] Chart of FAM FIS classification for SAVEE database --------------- [Figure 5-11] Chart of FIS vs. SVM classification for MFCC feature of SAVEE database ---::----::----::--++** 22 2S [Figure 5-12] Chart of FIS vs. SVM classification for GFCC feature of SAVEE database 00 a III 505000000 0 Ce ery [Figure 5-13] Chart of FIS vs.
SVM classification for GPLP feature of SAVEE database A RS > ee ee ene ee ea -viii- ABSTRACT Speech Emotion Recognition using Fuzzy Inference System based on Fuzzy Associative Memory TON THAT HOA AN Department of Digital Media Graduate School of Soongsil University Affective interaction is the high-level phase of human computer interaction. And Affective Computing is a terminology to describe a not long ago established active interdisciplinary research field dealing with the automatic sense, recognition and synthesis of human emotions from any biological modality such as speech or facial expression. Being one among research directions of Affective Computing, speech emotion recognition is a relatively recent research field which is defined as extracting the emotional state of a speaker from his or her speech. The paralinguistic information conveyed by speech emotions has been found to be useful in multiple ways such as extracting useful semantics from speech to improve the performance of speech recognition systems in speech processing or serving as an important ingredient of “emotional intelligence” of machines and contributing to human-machine interaction.
-ÌX- Despite speech emotion recognition has been investigated for past four decades but until now, there are still many difficulties and challenges. Two main problems in the research field are speech feature extraction and speech emotion classification. These are also two key research directions to improve the accuracy of speech emotion recognition systems. This dissertation presents a novel method of classification using fuzzy inference system based on fuzzy association memory.
To compare with method using support vector machine which is being used commonly at present, the work experimented on some existing single features as Mel-Frequency Cepstral Coefficients (MFCC), Greenwood Frequency Cepstral Coefficients (GFCC), Greenwood Perceptual Linear Prediction coefficients (GPLP). The classifiers have been experimented on two databases: Berlin Emotion Speech Database (Berlin Emo-DB) in German and Surrey Audio-Visual Expressed Emotion (SAVEE) database in English. Our experimental results show that the classifier using fuzzy inference system based on fuzzy association memory is better than the classification method using support vector machine on same kind of features and databases. The algorithms and experimental results of the classifier presented in the dissertation are only first steps.