||Development of an Automatic ECG-based Emotion Classification Algorithm
||Department of Electrical Engineering
real-time R-wave detection
automatic emotion classification algorithm
This thesis presents a real-time ECG morphology feature based R-wave detection algorithm and an automatic ECG-based emotion classification algorithm for R-wave detection and human emotion classification, respectively. At first, we adopt a musical induction method to collect participants’ ECG signals without any deliberate laboratory setting, which can induce participants’ real emotional states. Next, the proposed real-time R-wave detection algorithm is presented to detect R-waves in ECG signal based on the ECG morphological features. Afterward, we develop an automatic ECG-based emotion classification algorithm to classify human emotions elicited by listening to music. Physiological ECG features generated from time-, frequency-domain, and nonlinear analyses are utilized to find the emotion-relevant features and to correlate them with emotional states. Subsequently, we develop a sequential forward floating selection-kernel-based class separability based (SFFS-KBCS-based) feature selection algorithm and utilize the generalized discriminant analysis (GDA) to effectively select significant ECG features associated with emotions and to reduce the dimensions of the selected features, respectively. Classifications of positive/negative valence, high/low arousal, and four types of emotion (Joy, Tension, Sadness, and Peacefulness) are performed by least squares support vector machine (LS-SVM) classifiers. The results show that the sensitivity, positive predictive value, and detection error rate of the real-time R-wave detection algorithm can achieve 99.97%, 99.89%, and 0.14%, respectively, and the average delay time of the proposed algorithm is only 15.1ms. Furthermore, the correct classification rates of the positive/negative valence, high/low arousal, and four types of emotion classification tasks are 82.78%, 72.91%, and 61.52%, respectively.
Keywords: real-time R-wave detection, ECG, automatic emotion classification algorithm, musical induction.
CHINESE ABSTRACT i
TABLES OF CONTENTS v
LIST OF TABLES vii
LIST OF FIGURES ix
Chapter 1 Introduction 1
1.1 Motivation 1
1.2 Literature Survey 3
1.2.1 Dimensional Emotion Models 3
1.2.2 ECG Signal and Emotion 4
1.2.3 Music and Emotion 5
1.2.4 Approaches to Emotion Classification Using Biosignals 6
1.3 Purpose of the Study 7
1.4 Organization of the Thesis 8
Chapter 2 A Real-time ECG Morphology Feature Based R-wave Detection Algorithm 9
2.1 Introduction 10
2.2 ECG Morphology Feature Parameter Initialization 12
2.3 QR-wave Candidate Selection 13
2.4 Feature Generation 14
2.5 R-wave Detection 15
2.6 Dynamic Threshold Update Process 17
Chapter 3 Automatic ECG-based Emotion Classification Algorithm 18
3.1 Signal Preprocessing 20
3.1.1 Baseline Wander Removal 21
3.1.2 Z-score Normalization Method 22
3.2 R-wave Detection 23
3.3 Windowing 24
3.4 Incorrect Epoch Rejection 24
3.5 Feature Generation 25
3.5.1 Time-domain Analysis 25
3.5.2 Frequency-domain Analysis 27
3.5.3 Nonlinear Analysis 30
3.6 Feature Normalization 39
3.7 Feature Selection 39
3.8 Feature Extraction 44
3.8.1 Principal Component Analysis 44
3.8.2 Linear Discriminant Analysis 45
3.8.3 Generalized Discriminant Analysis 47
3.9 Classifier Construction 50
Chapter 4 Experimental Setup and Results 53
4.1 Experimental Setup 53
4.1.1 Materials and Setup 53
4.1.2 Experimental Protocol 54
4.1.3 Participant Self-assessment 56
4.2 Results of the Real-time R-wave Detection Algorithm 57
4.3 Results of the Automatic ECG-based Emotion Classification Algorithm 59
4.3.1 Positive/Negative Valence Classification 61
4.3.2 High/Low Arousal Classification 67
4.4 Discussions 72
4.4.1 Comparison of the Proposed Method with Other Existing Approach for the MAHNOB-HCI Database 72
4.4.2 Comparison of the Proposed Method with Other Existing Approaches Using Biosignals 73
Chapter 5 Conclusions and Future Work 75
5.1 Conclusions 75
5.2 Future Work 77
 U. R. Acharya, K. P. Joseph, N. Kannathal, C. M. Lim, and J. S. Suri, “Heart rate variability: A review,” Medical and Biological Engineering and Computing, vol. 44, pp. 1031-1051, 2006.
 G. Baudat and F. Anouar, “Generalized discriminant analysis using a kernel approach,” Neural Computation, vol. 12, no. 10, pp. 2385–2404, 2000.
 G.E.A.P.A. Batista, R.C. Prati, and M.C. Monard, “A study of the behavior of several methods for balancing machine learning training data,” ACM SIGKDD Explorations Newsletter, vol. 6, no. 1, pp. 20-29, 2004
 R. Bailón, L. Sörnmo, and P. Laguna, “A robust method for ECG-based estimation of the respiratory frequency during stress testing,” IEEE Trans. Biomedical Engineering, vol. 53, no. 7, pp. 1273-1285, 2006.
 S. Badiezadegan and S. Z. Hamid, “Design and evaluation of matched wavelets with maximum coding gain and minimum approximation error criteria for R peak detection in ECG,” International Journal of Wavelets, Multiresolution and Information Processing, vol. 6, pp. 799-825, 2008.
 P. de Chazal, C. Heneghan, E. Sheridan, R. Reilly, P. Nolan, and M. O’Malley, “Automated processing of the single-lead electrocardiogram for the detection of obstructive sleep apnea,” IEEE Trans. Biomedical Engineering, vol. 50, no. 6, pp. 686–696, 2003.
 I. Christov, “Real time electrocardiogram QRS detection using combined adaptive threshold,” Biomedical Engineering Online, vol. 3, pp. 28, 2004.
 W. B. Davis and M. H. Thaut, “The influence of preferred relaxing music on measures of state anxiety, relaxation, and physiological responses,” Journal of Music Therapy, vol. 26, no. 4, pp. 168-187, 1989.
 P. Ekman, “An argument for basic emotions,” Cognition and Emotion, vol. 6, pp. 169-200, 1992.
 C. E. Guzzetta, “Effects of relaxation and music therapy on patients in a coronary care unit with presumptive acute myocardial infarction,” Heart & Lung : The Journal of Critical Care, vol. 18, no. 6, pp. 609-616, 1989.
 A. L. Goldberger, A. N. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C. K. Peng, and H. E. Stanley, “PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signal,” Circulation, vol. 101, pp.215-220, 2000.
 Y. Gu, S. L. Tan, K. J. Wong, M. H. R. Ho, and L. Qu, “A biometric signature based system for improved emotion recognition using physiological responses from multiple subjects,” in Proc. of 8th IEEE Int’l Conf. Industrial Informatics, pp. 61-66, 2010.
 Y. C. Huang, S. H. Lin, C. Y. Chien, Y. C. Chen, L. C. Chou, S. C. Huang, and M. Y. Jan, “A biomedical entertainment platform design based on musical rhythm characteristic and heart rate variability (HRV),” IEEE Int’l Conf. Multimedia and Expo, pp. 385-388, 2008.
 C. Izard, The Face of Emotion, vol. 23. Appleton-Century-Crofts, 1971.
 N. Japkowicz and S. Stephen, “The class imbalance problem: A systematic study,” Intelligent Data Analysis, vol. 6, pp. 429-449, 2002.
 C. L. Krumhansl, “An exploratory study of musical emotions and psychophysiology,” Canadian Journal of Experimental Psychology, vol. 51, pp. 336-352, 1997.
 K. H. Kim, S. W. Bang, and S. R. Kim, “Emotion recognition system using short-term monitoring of physiological signals,” Medical & Biological Engineering & Computing, vol. 42, pp. 419-427, 2004.
 D. Kulić and A. Croft, “Affective state estimation for human-robot interaction,” IEEE Trans. Robotics, vol. 23, no. 5, pp. 991-1000, 2007.
 J. Kim and E. André, “Emotion recognition based on physiological changes in music listening,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 12, pp. 2067-2083, 2008.
 S. D. Kreibig, “Autonomic nervous system activity in emotion: A review,” Biological Psychology, vol. 84, pp. 394-421, 2010.
 S. Koelstra, C. Mühl, M. Soleymani, J. S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, “DEAP: A database for emotion analysis using physiological signals,” IEEE Trans. Affective computing, vol. 3, no. 1, 2012.
 P. J. Lang, “The emotion probe: Studies of motivation and attention,” American Psychologist,, vol. 50, no. 5, pp. 372-385, 1995.
 R, D, Lane, K. McRae, E. M. Reiman, K. Chen, G. L. Ahern, and J. F. Thayer, “ Neural correlates of heart rate variability during emotion,” NeuroImage, vol. 44, pp. 213-222, 2009.
 K. Z. Mao, K. C. Tan, and W. Ser, “Probabilistic neural-network structure determination for pattern classification,” IEEE Trans. Neural Networks, vol. 11, no. 4, pp. 1009-1016, 2000.
 J. McNames and M. Aboy, “Reliability and accuracy of heart rate variability metrics versus ECG segment duration,” Medical & Biological Engineering & Computing, vol. 44, pp. 747-756, 2006.
 M. Mneimneh , E. Yaz , M. Johnson, and R. Povinelli, “An adaptive kalman filter for removing baseline wandering in ECG signals,” in Proc. Computers in Cardiology, pp.253 -256, 2006.
 L. Y. D. Marco and L. Chiari, “A wavelet-based ECG delineation algorithm for 32-bit integer online processing,” Biomedical Engineering Online, 2011.
 J. P. Niskanen, M. P. Tarvainen, P. O. Ranta-aho, and P. A. Karjalainen, “Software for advanced HRV analysis,” Computer Methods and Programs in Biomedicine, vol. 76, pp. 73-81, 2004.
 C. O’Brien and C. Heneghan, “A comparison of algorithms for estimation of a respiratory signal from the surface electrocardiogram,” Computers in Biology and Medicine, vol. 37, pp. 305-314, 2007.
 M. Orini, R. Bailón, R. Enk, S. Koelsch, L. Mainardi, and P.Laguna, “A method for continuously assessing the automatic response to music-induced emotions through HRV analysis,” Medical & Biological Engineering & Computing, vol. 48, pp. 423-433, 2010.
 R. Plutchik, “Emotions: A general psychoevolutionary theory,” Approaches to Emotion, pp. 197-219, L. Erlbaum Assoc., 1984.
 J. Pan and W. J. Tompkins, “A real-time QRS detection algorithm,” IEEE Trans. Biomedical Engineering, vol. BME-32, pp. 230-236, 1985.
 P. Pudil, J. Novovičová, and J. Kittler, “Floating search methods in feature selection,” Pattern Recognition Letters, vol. 15, pp. 1119-1125, 1994.
 J. Posner, J. A. Russell, and B. S. Peterson, “The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology,” Development and Psychopathology, vol. 17, no. 3, pp. 715-734, 2005.
 S. B. Park, Y. S. Noh, S. J. Park, and H. R. Yoon, “An improved algorithm for respiration signal extraction from electrocardiogram measured by conductive textile electrodes using instantaneous frequency estimation,” Medical & Biological Engineering & Computing, vol. 46, pp. 147-158, 2008.
 J. A. Russell, “A circumplex model of affect,” Journal of Personality and Social Psychology, vol. 39, no. 6, pp. 1161-1178, 1980.
 A. Ruha, S. Sallinen, and S. Nissilä, “A real-time microprocessor QRS detector system with a 1-ms timing accuracy for the measurement of ambulatory HRV,” IEEE Trans. Biomedical Engineering, vol. 44, pp. 159-167, 1997.
 P. Rainville, A. Bechara, N. Naqvi, and A. R. Damasio, “Basic emotions are associated with distinct patterns of cardiorespiratory activity,” International Journal of Psychophysiology, vol. 61, pp. 5-18, 2006.
 G. Rigas, C. D. Katsis, G. Ganiatsas, and D. I. Fotiadis, “A user independent, biosignal based, emotion recognition method,” in Proc. 11th Int’l conf. User Modeling, pp. 314-318, 2007.
 H. Scholsberg, “Three dimensions of emotion,” Psychological Review, vol. 61, pp. 81-88, 1954.
 D. F. Specht, “Probabilistic neural network,” Neural Networks, vol. 3, pp. 109-118, 1990.
 J. A. K. Suykens and J. Vandewalle, “Least squares support vector machine classifiers,” Neural Processing Letters, vol. 9, pp. 293-300, 1999.
 A P. Sutarto, M. N. A. Wahab, and N. M. Zin, “Heart rate variability (HRV) biofeedback: A new training approach for operator's performance enhancement,” Journal of Industrial Engineering and Management, vol. 3, no. 1, pp. 176-198, 2010.
 M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A multimodal database for affect recognition and implicit tagging,” IEEE Trans. Affective computing, vol. 3, no. 1, 2012.
 S. Tompkins, Affect Imagery Consciousness: The Positive Affects, vol. 1. Springer Publishing Company, 1962.
 W. A. Tiller, R. McCraty, and M. Atkinson, “Cardiac coherence: A new, noninvasive measure of autonomic nervous system order,” Alternative Therapies, vol. 2, no. 1, pp. 52–65, 1996.
 M. P. Tulppo, T. H. Mäkikallio, T. E. S. Takala, T. Seppänen, and H. V. Huikuri, “Quantitative beat-to-beat analysis of heart rate dynamics during exercise,” American Journal of Physiology-Heart and Circulatory Physiology, vol. 271, pp. 244-252, 1996.
 S. Tabakov, I. Iliev, and V. Krasteva, “Online digital filter and QRS detector applicable in low resource ECG monitoring systems,” Annals of Biomedical Engineering, vol. 36, pp. 1805–1815, 2008.
 W. Trost, T. Ethofer, M. Zentner, and P. Vuilleumier, “Mapping aesthetic musical emotions in the brain,” Cerebral Cortex, vol. 22, no. 12, pp. 2769-2783, 2012.
 G. D. Vito, S. D. R. Galloway, M. A. Nimmo, P. Maas, and J. J. V. McMurray, “Effects of central sympathetic inhibition on heart rate variability during steady-state exercise in healthy humans,” Clinical Physiology and Functional Imaging, vol. 22, pp. 32-38, 2002.
 G. Valenza, A. Lanatà, and E. P. Scilingo, “Oscillations of heart rate and respiration synchronize during affective visual stimulation,” IEEE Trans. Information Technology in Biomedicine, vol. 16, no. 4, 2012.
 Z. S. Wang and J. D. Z. Chen, “Robust ECG R-R wave detection using evolutionary-programming-based fuzzy inference system (EPFIS), and application to assessing brain-gut interaction,” IEE Proceedings-Science, Measurement and Technology, vol. 147, pp. 351-356, 2000.
 L. Wang, “Feature selection with kernel class separability,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 9, pp. 1534-1546, 2008.
 Y. Wang, L. Li, J. Ni, and S. Huang, “Feature selection using tabu search with long-term memories and probabilistic neural networks,” Pattern Recognition Letters, vol. 30, pp. 661-670, 2009.
 J. S. Wnag, W. C. Chiang, Y. L. Hsu, and Y. T. C. Yang, “ECG arrhythmia classification using a probabilistic neural network with a feature reduction method,” Neurocomputing, vol. 116, pp. 38-45, 2013.
 M. Zentner, D. Grandjean, and K. R. Scherer, “Emotions evoked by the sound of music: Characterization, classification, and measurement,” Emotion, vol. 8, no. 4, pp. 494-521, 2008.
 Task Force of the European Society of Cardiology and North American Society of Pacing and Electrophysiology, “Heart rate variability: Standards of measurement, physiological interpretation and clinical use,” European Heart Journal, vol. 17, pp. 354–381, 1996.