系統識別號 U0026-1910201718314000
論文名稱(中文) 紅外熱序列臉部影像分析與應用
論文名稱(英文) The Analysis and Application of Facial Sequence Images with an Infrared Thermal Approach
校院名稱 成功大學
系所名稱(中) 航空太空工程學系
系所名稱(英) Department of Aeronautics & Astronautics
學年度 106
學期 1
出版年 106
研究生(中文) 簡伯霖
研究生(英文) Bo-Lin Jian
學號 P48011048
學位類別 博士
語文別 英文
論文頁數 89頁
口試委員 指導教授-陳介力
中文關鍵字 影像校準  獨立主成分分析  情緒評估  情緒顯著性地圖  熱影像 
英文關鍵字 Emotion Activation Map  Emotion Assessment  Independent Component Analysis  Image Registration  Thermal Image 
中文摘要 本論文使用國際情緒圖片系統挑選出不同的情緒圖片來誘發受試者的情緒且同步收集人臉紅外熱序列影像,將所收集到的序列影像自動校準並分析臉部溫度反應的狀況。ble首先,提出一套針對人臉紅外線熱序列影像校準流程,考量在不影響受試者舒適度與最小傷害原則的情況下,來降低受試者非意識的頭部晃動所產生之影像偏差。將一張熱影像藉由影像雙眼區域之質心點定位,來決定影像平移以及旋轉處理,產出校準用之固定影像。本研究提出兩階段基因演算法完成熱序列影像的自動校準,再以影像品質評估指標驗證來校準影像的前後差異。藉由上述之流程,進行精確的人臉影像對位;將自動校準後的序列影像進行人臉區域分割,針對每位受測者的前額、鼻子、嘴、左臉頰與右臉頰五個區域計算其平均溫度,以主成分分析方法將特徵值降維,再使用多變量變異數來分析其顯著性,並以支持向量機分類器來驗證中度與重度思覺失調症患者的臉部溫度差異。此結果與腦部額葉造成不對稱情緒理論相符,並且辨識率最高達94.3%。最後提出建構情緒顯著性地圖的演算法,以拆解出獨立的時間序列與對應的空間獨立成分,其中採用變異數分析計算情緒時序模板與時間序列的最大關聯性,此方法可視覺化顯示誘發的情緒與人臉溫度變化的顯著性。
英文摘要 In this paper, different types of emotion images, selected from the international affective picture system were presented to subjects during infrared thermal facial image approach in order to collect sequence images for further analysis and evaluation of changes in facial temperature. First, a set of facial infrared thermal sequence imaging procedures to reduce the image deviation caused by unconscious head shaking without compromising the subject's comfort and resulting in minimal damage is proposed. A thermal image is positioned by the center of mass of the image binocular area to determine the image translation and rotation processing and to output a fixed image for calibration. In this study, the automatic calibration of thermal sequence images was completed by the proposed two-stage gene algorithm, and the difference in images was calibrated by image quality evaluation. Through the above process the precise face images were aligned, the automatic calibration of the sequence image for facial area was segmented, and then the mean temperature of each subject's forehead, nose, mouth, left cheek and right cheek areas were calculated. The features was dimensioned by the principal component analysis method, the significance was analyzed by using the multivariate variance number, and the support vector machine classifier was used to verify the temperature difference of the face of patients with moderate and severe levels of schizophrenia. The results were consistent with the theory as the frontal lobe produced an asymmetrical emotional, with an identification rate of up to 94.3%. Finally, we propose an algorithm to construct the map of emotional significance to disassemble the independent time series and the corresponding independent spatial composition. Analysis of variance was used to calculate the maximum correlation between the emotion sequence template and the time series. This method can visualize the significant evoked emotional and facial temperature changes.
論文目次 摘要 I
1.1 Motivation 1
1.2 Literature Review 2
1.2.1 Infrared Thermal Image Sequence Registration 2
1.2.2 Emotion evoked methods 4
1.2.3 Emotion Recognition 5
1.2.4 Independent Component Analysis of Activation Maps 7
1.3 Structure of the Dissertation 8
2.1 Participants 10
2.2 Experimental Setup 13
2.3 Stimuli and Paradigm 15
3.1 Calibration of the fixed image 17
3.2 Registration of thermal facial sequences 22
3.3 Discussion 29
4.1 Features Extraction and Analysis 38
4.2 MANOVA of Facial Areas in Response to Evoked Emotions 39
4.3 SVM Identification Results using Different Numbers of Features 41
4.4 Discussions 43
5.1 Process of Regional Activation and Construction of the Activation Map 48
5.1.1 Emotion Activation Maps 50
5.1.2 Facial Regions 54
5.2 Discussion 55
參考文獻 [1] Beham, M. P., and Roomi, S. M. M., 2013, "A Review of Face Recognition Methods," Int. J. Pattern Recognit. Artif. Intell., 27(4), p. 35.
[2] Gade, R., and Moeslund, T. B., 2014, "Thermal cameras and applications: a survey," Machine Vision and Applications, 25(1), pp. 245-262.
[3] Santhanaganesh, A. S., and Rajakumar, P. S., 2014, "Facial Expression Recognition in Various Illuminous Environment," Digital Image Processing, 6(3).
[4] Ioannou, S., Gallese, V., and Merla, A., 2014, "Thermal infrared imaging in psychophysiology: potentialities and limits," Psychophysiology, 51(10), pp. 951-963.
[5] Ioannou, S., Morris, P., Mercer, H., et al., 2014, "Proximity and gaze influences facial temperature: a thermal infrared imaging study," Front Psychol, 5, p. 845.
[6] Di Giacinto, A., Brunetti, M., Sepede, G., et al., 2014, "Thermal signature of fear conditioning in mild post traumatic stress disorder," Neuroscience, 266, pp. 216-223.
[7] Manini, B., Cardone, D., Ebisch, S. J., et al., 2013, "Mom feels what her child feels: thermal signatures of vicarious autonomic response while watching children in a stressful situation," Frontiers in human neuroscience, 7, p. 299.
[8] Rajoub, B. A., and Zwiggelaar, R., 2014, "Thermal Facial Analysis for Deception Detection," Ieee Transactions on Information Forensics and Security, 9(6), pp. 1015-1023.
[9] Esposito, G., Nakazawa, J., Ogawa, S., et al., 2015, "Using infrared thermography to assess emotional responses to infants," Early Child Development and Care, 185(3), pp. 438-447.
[10] Costello, J. T., McInerney, C. D., Bleakley, C. M., et al., 2012, "The use of thermal imaging in assessing skin temperature following cryotherapy: a review," Journal of Thermal Biology, 37(2), pp. 103-110.
[11] Lahiri, B. B., Bagavathiappan, S., Jayakumar, T., et al., 2012, "Medical applications of infrared thermography: A review," Infrared Physics & Technology, 55(4), pp. 221-235.
[12] Mostafa, E., Hammoud, R., Ali, A., et al., 2013, "Face recognition in low resolution thermal images," Computer Vision and Image Understanding, 117(12), pp. 1689-1694.
[13] Budzan, S., and Wyzgolik, R., 2013, "Face and eyes localization algorithm in thermal images for temperature measurement of the inner canthus of the eyes," Infrared Physics & Technology, 60(0), pp. 225-234.
[14] Torabi, A., and Bilodeau, G. A., 2013, "Local self-similarity-based registration of human ROIs in pairs of stereo thermal-visible videos," Pattern Recognition, 46(2), pp. 578-589.
[15] Eveland, C. K., Socolinsky, D. A., and Wolff, L. B., 2003, "Tracking human faces in infrared video," Image and Vision Computing, 21(7), pp. 579-590.
[16] Shoja Ghiass, R., Arandjelović, O., Bendada, A., et al., 2014, "Infrared face recognition: A comprehensive review of methodologies and databases," Pattern Recognition, 47(9), pp. 2807-2824.
[17] Tsai, C. L., Li, C. Y., Yang, G., et al., 2010, "The edge-driven dual-bootstrap iterative closest point algorithm for registration of multimodal fluorescein angiogram sequence," IEEE Trans Med Imaging, 29(3), pp. 636-649.
[18] Zhilkin, P., and Alexander, M. E., 2004, "Affine registration: a comparison of several programs," Magn Reson Imaging, 22(1), pp. 55-66.
[19] Gross, J. J., and Levenson, R. W., 1995, "Emotion Elicitation Using Films," Cognition & Emotion, 9(1), pp. 87-108.
[20] Lang, P. J., Simons, R. F., and Balaban, M., 2013, Attention and orienting: Sensory and motivational processes, Psychology Press.
[21] Palomba, D., Sarlo, M., Angrilli, A., et al., 2000, "Cardiac responses associated with affective processing of unpleasant film stimuli," Int J Psychophysiol, 36(1), pp. 45-57.
[22] Fu, C. H., Williams, S. C., Cleare, A. J., et al., 2004, "Attenuation of the neural response to sad faces in major depression by antidepressant treatment: a prospective, event-related functional magnetic resonance imaging study," Arch Gen Psychiatry, 61(9), pp. 877-889.
[23] Guntekin, B., and Basar, E., 2014, "A review of brain oscillations in perception of faces and emotional pictures," Neuropsychologia, 58, pp. 33-51.
[24] Jerram, M., Lee, A., Negreira, A., et al., 2014, "The neural correlates of the dominance dimension of emotion," Psychiatry Res, 221(2), pp. 135-141.
[25] Kukolja, D., Popovic, S., Horvat, M., et al., 2014, "Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications," International Journal of Human-Computer Studies, 72(10-11), pp. 717-727.
[26] Gaetano, V., Antonio, L., and Enzo Pasquale, S., 2013, "Improving emotion recognition systems by embedding cardiorespiratory coupling," Physiological Measurement, 34(4), p. 449.
[27] Kleinsmith, A., and Bianchi-Berthouze, N., 2013, "Affective Body Expression Perception and Recognition: A Survey," Ieee T Affect Comput, 4(1), pp. 15-33.
[28] Beaudry, O., Roy-Charland, A., Perron, M., et al., 2014, "Featural processing in recognition of emotional facial expressions," Cognition & Emotion, 28(3), pp. 416-432.
[29] Wang, Z. F., Miao, Z. J., Wu, Q. M. J., et al., 2014, "Low-resolution face recognition: a review," Visual Comput, 30(4), pp. 359-386.
[30] Ulukaya, S., and Erdem, C. E., 2014, "Gaussian mixture model based estimation of the neutral face shape for emotion recognition," Digital Signal Processing, 32, pp. 11-23.
[31] Azazi, A., Lotfi, S. L., Venkat, I., et al., 2015, "Towards a robust affect recognition: Automatic facial expression recognition in 3D faces," Expert Systems with Applications, 42(6), pp. 3056-3066.
[32] BEHAM, M. P., and ROOMI, S. M. M., 2013, "A REVIEW OF FACE RECOGNITION METHODS," International Journal of Pattern Recognition and Artificial Intelligence, 27(04), p. 1356005.
[33] Ring, E. F., and Ammer, K., 2012, "Infrared thermal imaging in medicine," Physiol Meas, 33(3), pp. R33-46.
[34] Pollina, D. A., Dollins, A. B., Senter, S. M., et al., 2006, "Facial skin surface temperature changes during a "concealed information" test," Ann Biomed Eng, 34(7), pp. 1182-1189.
[35] Nhan, B. R., and Chau, T., 2009, "Infrared thermal imaging as a physiological access pathway: a study of the baseline characteristics of facial skin temperatures," Physiol Meas, 30(4), pp. N23-35.
[36] Nhan, B. R., and Chau, T., 2010, "Classifying affective states using thermal infrared imaging of the human face," IEEE Trans Biomed Eng, 57(4), pp. 979-987.
[37] Shastri, D., Merla, A., Tsiamyrtzis, P., et al., 2009, "Imaging facial signs of neurophysiological responses," IEEE Trans Biomed Eng, 56(2), pp. 477-484.
[38] Gane, L., Power, S., Kushki, A., et al., 2011, "Thermal imaging of the periorbital regions during the presentation of an auditory startle stimulus," PLoS One, 6(11), p. e27268.
[39] Perry, A., Aviezer, H., Goldstein, P., et al., 2013, "Face or body? Oxytocin improves perception of emotions from facial expressions in incongruent emotional body context," Psychoneuroendocrinology, 38(11), pp. 2820-2825.
[40] Lischke, A., Berger, C., Prehn, K., et al., 2012, "Intranasal oxytocin enhances emotion recognition from dynamic facial expressions and leaves eye-gaze unaffected," Psychoneuroendocrinology, 37(4), pp. 475-481.
[41] Hyvarinen, A., and Oja, E., 2000, "Independent component analysis: algorithms and applications," Neural Netw, 13(4-5), pp. 411-430.
[42] Chen, K., Chen, X., Renaut, R., et al., 2007, "Characterization of the image-derived carotid artery input function using independent component analysis for the quantitation of [18F] fluorodeoxyglucose positron emission tomography images," Phys Med Biol, 52(23), pp. 7055-7071.
[43] Kuan-Hao, S., Jih-Shian, L., Jia-Hung, L., et al., 2009, "Partial volume correction of the microPET blood input function using ensemble learning independent component analysis," Physics in Medicine and Biology, 54(6), p. 1823.
[44] Chen, R.-B., Chen, Y., and Härdle, W. K., 2014, "TVICA—Time varying independent component analysis and its application to financial data," Computational Statistics & Data Analysis, 74, pp. 95-109.
[45] Beckmann, C. F., 2012, "Modelling with independent components," Neuroimage, 62(2), pp. 891-901.
[46] Hyvarinen, A., 1999, "Fast and robust fixed-point algorithms for independent component analysis," IEEE Trans Neural Netw, 10(3), pp. 626-634.
[47] Suzuki, K., Kiryu, T., and Nakada, T., 2002, "Fast and precise independent component analysis for high field fMRI time series tailored using prior information on spatiotemporal structure," Human brain mapping, 15(1), pp. 54-66.
[48] Shen, H., Kleinsteuber, M., and Huper, K., 2008, "Local convergence analysis of FastICA and related algorithms," IEEE Trans Neural Netw, 19(6), pp. 1022-1032.
[49] Furukawa, T. A., Levine, S. Z., Tanaka, S., et al., 2015, "Initial Severity of Schizophrenia and Efficacy of Antipsychotics Participant-Level Meta-analysis of 6 Placebo-Controlled Studies," Jama Psychiatry, 72(1), pp. 14-21.
[50] Leucht, S., Kane, J. M., Kissling, W., et al., 2005, "What does the PANSS mean?," Schizophrenia Research, 79(2-3), pp. 231-238.
[51] Chen, Y. T., Huang, M. W., Hung, I. C., et al., 2014, "Right and left amygdalae activation in patients with major depression receiving antidepressant treatment, as revealed by fMRI," Behavioral and Brain Functions, 10.
[52] Assoc, W. M., 2013, "World Medical Association Declaration of Helsinki Ethical Principles for Medical Research Involving Human Subjects," Jama-Journal of the American Medical Association, 310(20), pp. 2191-2194.
[53] Lang, P. J., Bradley, M. M., and Cuthbert, B. N., 2008, "International affective picture system (IAPS): Affective ratings of pictures and instruction manual."
[54] Lang, P. J., Bradley, M. M., and Cuthbert, B. N., 1997, "Motivated attention: Affect, activation, and action," Attention and orienting: Sensory and motivational processes, pp. 97-135.
[55] Guntekin, B., and Basar, E., 2014, "A review of brain oscillations in perception of faces and emotional pictures," Neuropsychologia, 58(1), pp. 33-51.
[56] Jerram, M., Lee, A., Negreira, A., et al., 2014, "The neural correlates of the dominance dimension of emotion," Psychiatry Research-Neuroimaging, 221(2), pp. 135-141.
[57] Chen, C. L., and Jian, B. L., 2015, "Infrared thermal facial image sequence registration analysis and verification," Infrared Phys Techn, 69, pp. 1-6.
[58] Tsai, C. L., Li, C. Y., and Yang, G., 2010, "Lin KS: : the edge-driven dual-bootstrap iterative closest point algorithm for registration of multimodal fluorescein angiogram sequence," IEEE Trans Med Imaging, 29.
[59] Goshtasby, A. A., 2012, Image Registration: Principles, Tools and Methods, Springer.
[60] Bilodeau, G.-A., Torabi, A., St-Charles, P.-L., et al., 2014, "Thermal–visible registration of human silhouettes: A similarity measure performance evaluation," Infrared Physics & Technology, 64(0), pp. 79-86.
[61] Ye, Y. X., and Shan, J., 2014, "A local descriptor based registration method for multispectral remote sensing images with non-linear intensity differences," Isprs Journal of Photogrammetry and Remote Sensing, 90(0), pp. 83-95.
[62] Hu, S. B., and Shao, P., 2012, "Improved nearest neighbor interpolators based on confidence region in medical image registration," Biomedical Signal Processing and Control, 7(5), pp. 525-536.
[63] Studholme, C., Hill, D. L. G., and Hawkes, D. J., 1999, "An overlap invariant entropy measure of 3D medical image alignment," Pattern Recognition, 32(1), pp. 71-86.
[64] Rivaz, H., Karimaghaloo, Z., and Collins, D. L., 2014, "Self-similarity weighted mutual information: a new nonrigid image registration metric," Med Image Anal, 18(2), pp. 343-358.
[65] Holland, J. H., 1992, Adaptation in natural and artificial systems, MIT Press.
[66] Bautista, M. A., Escalera, S., Baro, X., et al., 2014, "On the design of an ECOC-Compliant Genetic Algorithm," Pattern Recognition, 47(2), pp. 865-884.
[67] Xie, F. Y., and Bovik, A. C., 2013, "Automatic segmentation of dermoscopy images using self-generating neural networks seeded by genetic algorithm," Pattern Recognition, 46(3), pp. 1012-1019.
[68] Bagheri, M. A., Gao, Q. G., and Escalera, S., 2013, "A genetic-based subspace analysis method for improving Error-Correcting Output Coding," Pattern Recognition, 46(10), pp. 2830-2839.
[69] Kumari, M. S., Priyanka, G., and Sydulu, M., "Comparison of genetic algorithms and particle swarm optimization for optimal power flow including FACTS devices," Proc. Power Tech, 2007 IEEE Lausanne, IEEE, pp. 1105-1110.
[70] Wang, Z., Leung, C. S., Wong, T. T., et al., 2004, "Eigen-image based compression for the image-based relighting with cascade recursive least squared networks," Pattern Recognition, 37(6), pp. 1219-1231.
[71] Wang, R. Z., Lin, C. F., and Lin, J. C., 2001, "Image hiding by optimal LSB substitution and genetic algorithm," Pattern Recognition, 34(3), pp. 671-683.
[72] Malik, A. S., and Choi, T. S., 2008, "A novel algorithm for estimation of depth map using image focus for 3D shape recovery in the presence of noise," Pattern Recognition, 41(7), pp. 2200-2225.
[73] Hu, X. F., and Weng, Q. H., 2009, "Estimating impervious surfaces from medium spatial resolution imagery using the self-organizing map and multi-layer perceptron neural networks," Remote Sensing of Environment, 113(10), pp. 2089-2102.
[74] Bayraktar, B., Bernas, T., Robinson, J. P., et al., 2007, "A numerical recipe for accurate image reconstruction from discrete orthogonal moments," Pattern Recognition, 40(2), pp. 659-669.
[75] Hill, D. L., Batchelor, P. G., Holden, M., et al., 2001, "Medical image registration," Phys Med Biol, 46(3), pp. R1-45.
[76] Vos, T., Flaxman, A. D., Naghavi, M., et al., 2012, "Years lived with disability (YLDs) for 1160 sequelae of 289 diseases and injuries 1990–2010: a systematic analysis for the Global Burden of Disease Study 2010," The Lancet, 380(9859), pp. 2163-2196.
[77] Hu, Z., Yang, W., Liu, H., et al., 2014, "From PET/CT to PET/MRI: advances in instrumentation and clinical applications," Mol Pharm, 11(11), pp. 3798-3809.
[78] Vidal, R., Ma, Y., and Sastry, S., 2005, "Generalized principal component analysis (GPCA)," IEEE Trans Pattern Anal Mach Intell, 27(12), pp. 1945-1959.
[79] Joachims, T., 1998, "Text categorization with support vector machines: Learning with many relevant features," Machine learning: ECML-98, pp. 137-142.
[80] Puri, C., Olson, L., Pavlidis, I., et al., "StressCam: non-contact measurement of users' emotional states through thermal imaging," Proc. CHI'05 extended abstracts on Human factors in computing systems, ACM, pp. 1725-1728.
[81] Zhu, Z., Tsiamyrtzis, P., and Pavlidis, I., "Forehead thermal signature extraction in lie detection," Proc. Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE, IEEE, pp. 243-246.
[82] Engert, V., Merla, A., Grant, J. A., et al., 2014, "Exploring the use of thermal infrared imaging in human stress research," PLoS One, 9(3), p. e90782.
[83] Shastri, D., Papadakis, M., Tsiamyrtzis, P., et al., 2012, "Perinasal Imaging of Physiological Stress and Its Affective Potential," Ieee T Affect Comput, 3(3), pp. 366-378.
[84] Ioannou, S., Ebisch, S., Aureli, T., et al., 2013, "The autonomic signature of guilt in children: a thermal infrared imaging study," PLoS One, 8(11), p. e79440.
[85] Davidson, R. A., Fedio, P., Smith, B. D., et al., 1992, "Lateralized mediation of arousal and habituation: differential bilateral electrodermal activity in unilateral temporal lobectomy patients," Neuropsychologia, 30(12), pp. 1053-1063.
[86] Wheeler, R. E., Davidson, R. J., and Tomarken, A. J., 1993, "Frontal brain asymmetry and emotional reactivity: a biological substrate of affective style," Psychophysiology, 30(1), pp. 82-89.
[87] Calhoun, V. D., and Adali, T., 2006, "Unmixing fMRI with independent component analysis," IEEE Eng Med Biol Mag, 25(2), pp. 79-90.
[88] Dennis, E. L., and Thompson, P. M., 2014, "Functional brain connectivity using fMRI in aging and Alzheimer's disease," Neuropsychol Rev, 24(1), pp. 49-62.
[89] Vargas, C., Lopez-Jaramillo, C., and Vieta, E., 2013, "A systematic literature review of resting state network--functional MRI in bipolar disorder," J Affect Disord, 150(3), pp. 727-735.
[90] Pollina, D. A., Dollins, A. B., Senter, S. M., et al., 2006, "Facial skin surface temperature changes during a "Concealed Information" test," Annals of Biomedical Engineering, 34(7), pp. 1182-1189.
  • 同意授權校內瀏覽/列印電子全文服務,於2022-09-25起公開。
  • 同意授權校外瀏覽/列印電子全文服務,於2022-09-25起公開。

  • 如您有疑問,請聯絡圖書館