The Use of Spectral Information in the Development of Novel Techniques for Speech- Based Cognitive Load Classification A thesis submitted for the degree of Doctor of Philosophy By Phu Ngoc Le Supervisor: Prof. Eliathamby Ambikairajah Co-supervisors: Dr. Julien Epps Dr. Eric Choi School of Electrical Engineering and Telecommunications The University of New South Wales January 2012 Abstract The cognitive load of a user refers to the amount of mental demand imposed on the user when performing a particular task.
Estimating the cognitive load (CL) level of the users is necessary to adjust the workload imposed on them accordingly in order to improve task performance. The current speech based CL classification systems are not adequate for commercial use due to their low performance particularly in noisy environments. This thesis proposes many techniques to improve the performance of the speech based cognitive load classification system in both clean and noisy conditions. This thesis analyses and presents the effectiveness of speech features such as spectral centroid frequency (SCF) and spectral centroid amplitude (SCA) for CL classification.
Sub-systems based on SCF and SCA features were developed and fused with the traditional Mel frequency cepstral coefficients (MFCC) based system, producing an 8.5% relative error rate reduction respectively when compared to the MFCC-based system alone. The Stroop test corpus was used in these experiments. The investigation into cognitive load information in the form of spectral distribution in different subbands shows that the information distributed in the low frequency subband is significantly higher than the high frequency subband. Two different methods are proposed to utilize this finding.
The first method, called the multi-band approach, uses a weighting scheme to emphasize the speech features in low frequency subbands. The cognitive load classification accuracy of this approach is shown to be higher than a system based on a non-weighting scheme. The second method is to design an effective filterbank based on the spectral distribution of cognitive load information using the Kullback-Leibler distance measure. It is shown that the designed filterbank consistently provides higher classification accuracies than other existing filterbanks such as mel, Bark, and equivalent rectangular bandwidth.
A discrete cosine transform based speech enhancement technique is proposed in order to increase the robustness of the CL classification system and found to be more suitable than other methods investigated. This proposed method provides a 3.0% average relative error rate reduction for the seven types of noise and five levels of SNR used. In particular, it provides a maximum of 7.5% relative error rate reduction for the F16 noise (in NOISEX-92 database) at 20 dB SNR. Keywords: Automatic cognitive load classification, cognitive load information distribution, filterbank designing, multi-band, weighting, speech enhancement.
i Acknowledgements I would like to express my sincere thanks to my supervisor Professor Eliathamby Ambikairajah for his invaluable guidance, encouragement, and technical support. I would also like to thank to my co-supervisors, Dr. Eric Choi and Dr. Julien Epps for their technical support and help in revising and correcting my technical writing.
From our speech research group, I would like to thank Dr. Vidhyasaharan Sethu and Dr. Tharmarajah Thiruvaran for many valuable discussions as well as their help in proof reading my thesis. I would also like to thank Dr.
Mohaddesh Nosratighods, Dr. Bo Yin, Dr. Teddy Gunawan for many technical discussions and valuable suggestions. I wish to thank Mr Tet Yap and Ms Karen Kua for their help in proof reading some parts of my thesis.
I would like to extend my thanks to other members of our research group, Dr. Mahmood Akhtar, Dr. Liang Wang, Dr. Ning Wang, Dr.
Ronny Kurniawan, and Ms Phyu Khing for their support. I would also like to thank all members of the Image Signal and Information Processing group at UNSW for their friendship and thank Mr. Tom Millet for organizing a warm and friendly working environment for us. I would like to thank Ms Raji Ambikairajah and Ms Stefanie Brown for their assistance in editing and proof reading this thesis.
I wish to acknowledge the Vietnamese government for funding my research. I also wish to acknowledge the National Information Communication Technology Australia (NICTA) and Graduate Research School at UNSW for the additional funding they provided. This research would not have been possible without all of this financial support. I also wish to thank the School of Electrical Engineering and Telecommunications at UNSW for providing me with travel support to attend conferences.
I wish to acknowledge the International Research Center Multimedia Information Communication and Application (MICA), Vietnam for giving me an opportunity to visit and work for a short-term at their center during my internship. Finally, I would like to express my sincere thanks to my parents, L. Cam, and my sister, L. Tai for their endless love, support and encouragement.
ii List of publications Journal paper 1. Choi, (2011) “Investigation of spectral centroid features for cognitive load classification”, Speech Communication, Vol. 53, Issue 4, April 2011, pp 540-551 Conference papers 1., (2011) “Investigation of the Robustness of a Non-Uniform Filterbank for Cognitive Load Classification”, in Proc. of the 8th International Conference on Information and Comunication System (ICICS) Singapore, Dec.
Sethu, (2010) “Robust Speech-Based Cognitive Load Classification Using a Multi-band Approach”, in Proc. of the Second APSIPA Annual Summit and Conference, Biopolis, Singapore, 2010, pp 400-404. Ambikairajah, (2010) "A study of voice source and vocal tract filter based features in cognitive load classification," in Proc. of the 20th International Conference on Pattern Recognition, Istanbul Turkey, 2010, pp 4516-4519.
Epps, (2009) “A Non-Uniform Subband Approach to Speech-Based Cognitive Load Classification” in Proc. of the 7th International Conference on Information and Comunication System (ICICS), Macau, Dec. Sethu, (2008) “Speech Enhancement Based On Empirical Mode Decomposition”, in Proc. of the IASTED International Conference on Signal Processing, Pattern Recognition and Applications, February 2008, at Innsbruck, Austria, pp.
Choi, (2008) "An Improved Soft Threshold Method for DCT Speech Enhancement", in Proc. of the Second International Conference on Communication and Electronics, Hoian, Vietnam 2008, pp 268 - 271. Ambikairajah, (2007) “Non-Uniform Sub-Band Kalman Filtering for Speech Enhancement”, in Proc. of International Conference on Signal Processing and Communication System (ICSPCS), Gold coast Australia, 2007.
Choi, (2009) “Improvement of Vietnamese Tone Classification using FM and MFCC Features”, presented at the IEEE-RIVF International Conference on Computing and Communication Technologies, Danang, Vietnam 2009, pp 140-143. iv Acronyms and Abbreviations AR Autoregressive CL Cognitive load DCT Discrete Cosine Transform EMD Empirical Mode Decomposition ERB Equivalent Rectangular Bandwidth FF Formant frequency FFT Fast Fourier Transform FM Frequency Modulation FMFCC Filter Mel Frequency Cepstral Coefficients GD Group Delay GMM Gaussian Mixture Model KL Kullback-Leibler IMF Intrinsic Mode Function MAP Maximum A Posteriori MFCC Mel Frequency Cepstral Coefficients PESQ Perceptual Evaluation of Speech Quality SCF Spectral Centroid Frequency SCA Spectral Centroid Amplitude SDF Shifted Delta Feature SI Spectral Intercept SMFCC Source Mel Frequency Cepstral Coefficients SNR Signal to Noise Ratio SS Spectral Slope SVM Support Vector Machines UBM Universal Background Model v Contents Abstract .1 Speech based cognitive load classification .3 Organization of the thesis. Chapter 2: Automatic cognitive load classification system .1 Working memory and its limitation .2 Cognitive load theory.3 Types of cognitive load .2 Overview of cognitive load measurement .1 Subjective or self-reporting measures .3 Cognitive load and speech.1 Effect of cognitive load variation on high-level speech features .2 Human speech production.3 Effect of cognitive load variation on low-level speech features .4 Automatic speech-based cognitive load classification system .1 Gaussian mixture model .3 Existing CL classification systems .5 Cognitive load speech corpora .1 Collection of the Stroop test database .2 Collection of the Reading and Comprehension database. Chapter 3: Investigation of the effectiveness of speech features for cognitive load classification .1 Source-filter model of human speech production system .1 The source component .2 The filter component .3 Combining the source and the filter components .2 Human listening test .2 Results and discussion .3 Speech cues of cognitive load .3 Baseline cognitive load classification system .2 Allocation of training and testing data .4 The effectiveness of source and filter based features .1 Source-based features .3 Source Mel frequency cepstral coefficients (SMFCC) .2 Filter-based features .2 Filter Mel frequency cepstral coefficients (FMFCC) .1 Mel frequency cepstral coefficients (MFCCs) .2 Spectral slope and spectral intercept .3 Group delay feature (GD).5 The effectiveness of spectral centroid features .2 Complementary behavior between spectral centroid and MFCC features .3 Cognitive load (CL) discrimination ability of spectral centroid features .4 Performance of the spectral centroid features .6 Comparison and discussion of performance of different speech features.
Chapter 4: Multi-band approach for cognitive load classification .2 Motivation for using a multi-band approach .1 Advantage of multi-band over full-band approach .1 Effect of band-limited noise .2 Effect of different types of noise .2 Variation of CL information in different subbands .1 Subband based feature extraction.2 Distribution of CL information in different mel subbands .3 Multi-band classification system .1 Overview of multi-band system .2 Classification experiment setup for multi-band approach .3 Estimation of weighting coefficients for likelihood combination .4 Performance of multi-band approach in clean condition .5 Performance of multi-band approach under noisy conditions .1 Reliability of subband speech features .2 Weighting schemes for likelihood combination .3 Comparison of the effectiveness of multi-band and full-band approaches.4 Performance of the multi-band system based on three subbands. Chapter 5: Investigation of cognitive load information distribution and filterbank design.2 The effect of varying the feature dimension of the spectral features .2 System performance with different feature dimensions .3 Evaluation of the correlation of SCF and SCA .3 The distribution of CL information across different frequency bands .1 Analysis on cepstral coefficients .1 Feature-based measure .2 Model-based measure .3 Performance based measure .2 Results from the analysis on SCF, SCA, and energy .3 Spectral distribution of CL information .4 Filterbank design for CL classification .1 Procedure to allocate center frequencies and bandwidths of the filters .2 Designing filterbank to extract cepstral coefficients .2 Performance of the designed filterbanks .3 Designing a filterbank to extract spectral centroid features .2 Performance of the designed filterbanks .4 Performance of designed filterbanks in noisy conditions. Chapter 6: Speech enhancement for cognitive load classification .2 Proposed speech enhancement methods .1 Kalman filtering method .1 Kalman filtering for speech enhancement.2 Traditional full-band Kalman filtering method .3 Proposed non-uniform subband Kalman filtering .2 Empirical mode decomposition based method .1 Empirical mode decomposition .2 Proposed speech enhancement method based on empirical mode decomposition .3 Speech enhancement in DCT domain .1 Traditional soft thresholding method .2 Proposed improved soft thresholding method .4 Comparison of the proposed speech enhancement methods.3 Incorporating the thresholding DCT module into CL classification system. Chapter 7: Conclusion and Future work .1 Implementation of human listening test .2 The use of spectral based speech features.3 Analysis of the distribution of cognitive load information .4 Multi-band approach and the effectiveness of weighting schemes .5 Designing effective filterbanks to extract spectral features .6 Proposed speech enhancement methods.
145 x List of Figures Figure 2.1: An illustration of three types of CL on working memory.2: Examples of 9-point and 7-point self-report rating scales.3: Speech production process [48].4: The diagram of an automatic speech-based CL classification system.5: Shifted delta feature calculation for a single feature stream at nth frame [60].6: Concatenation of the static and shifted delta features.7: The distribution of a speech feature before warping (a) & (b) and after warping (c) & (d).8: (a) Probability distribution of a single-dimensional feature, .9: Block diagram of an UBM-GMM based CL classification system .10: Overview of a CL classification system based on fusion technique.11: An example of two tasks of the Sroop test .2: The source-filter model for voiced speech production.3: Glottal filter model.4: (a) Magnitude spectrum of phoneme /i/, (b) the corresponding magnitude response of the vocal tract filter, (c) the corresponding magnitude spectrum of the glottal waveform.5: The listening test user interface.6: Accuracies of individual listener in the listening test.7: Allocation of training and testing speech data .8: Distribution of the pitch of the words ‘gray’.9: Block diagram of SMFCCs extraction.