Effective Facial Expression Recognition Through Multimodal Imaging for Traumatic Brain Injured Patient’s Rehabilitation

Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

Abstract

This article presents the issues related to applying computer vision techniques to identify facial expressions and recognize the mood of Traumatic Brain Injured (TBI) patients in real life scenarios. Many TBI patients face serious problems in communication and activities of daily living. These are due to restricted movement of muscles or paralysis with lesser facial expression along with non-cooperative behaviour, and inappropriate reasoning and reactions. All these aforementioned attributes contribute towards the complexity of the system for the automatic understanding of their emotional expressions. Existing systems for facial expression recognition are highly accurate when tested on healthy people in controlled conditions. However, their performance is not yet verified on the TBI patients in the real environment. In order to test this, we devised a special arrangement to collect data from these patients. Unlike the controlled environment, it was very challenging because these patients have large pose variations, poor attention and concentration with impulsive behaviours. In order to acquire high-quality facial images from videos for facial expression analysis, effective techniques of data preprocessing are applied. The extracted images are then fed to a deep learning architecture based on Convolution Neural Network (CNN) and Long Short-Term Memory (LSTM) network to exploit the spatiotemporal information with 3D face frontalization. RGB and thermal imaging modalities are used and the experimental results show that better quality of facial images and larger database enhance the system performance in facial expressions and mood recognition of TBI patients under natural challenging conditions. The proposed approach hopefully facilitates the physiotherapists, trainers and caregivers to deploy fast rehabilitation activities by knowing the positive mood of the patients.

Original languageEnglish
Title of host publicationComputer Vision, Imaging and Computer Graphics Theory and Applications : 13th International Joint Conference, VISIGRAPP 2018 Funchal–Madeira, Portugal, January 27–29, 2018, Revised Selected Papers
EditorsDominique Bechmann, Manuela Chessa, Ana Paula Cláudio, Francisco Imai, Andreas Kerren, Paul Richard, Alexandru Telea, Alain Tremeau
Number of pages21
PublisherSpringer Publishing Company
Publication date2 Jul 2019
Pages369-389
ISBN (Print)978-3-030-26755-1
ISBN (Electronic)978-3-030-26756-8
DOIs
Publication statusPublished - 2 Jul 2019
Event14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (Visigrapp 2019) - Prague, Czech Republic
Duration: 25 Feb 201927 Feb 2019

Conference

Conference14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (Visigrapp 2019)
CountryCzech Republic
CityPrague
Period25/02/201927/02/2019
SeriesCommunications in Computer and Information Science
VolumeCCIS, volume 997
ISSN1865-0929

Keywords

  • Computer vision
  • Convolution Neural Networks
  • Face detection
  • Facial Expressions Recognition
  • Facial landmarks
  • Long-Short Term Memory
  • Multi-visual (RGB, thermal) modalities
  • Traumatic Brain Injured Patients

Fingerprint Dive into the research topics of 'Effective Facial Expression Recognition Through Multimodal Imaging for Traumatic Brain Injured Patient’s Rehabilitation'. Together they form a unique fingerprint.

  • Projects

    Cite this

    Ilyas, C. M. A., Haque, M. A., Rehm, M., Nasrollahi, K., & Moeslund, T. B. (2019). Effective Facial Expression Recognition Through Multimodal Imaging for Traumatic Brain Injured Patient’s Rehabilitation. In D. Bechmann, M. Chessa, A. P. Cláudio, F. Imai, A. Kerren, P. Richard, A. Telea, & A. Tremeau (Eds.), Computer Vision, Imaging and Computer Graphics Theory and Applications: 13th International Joint Conference, VISIGRAPP 2018 Funchal–Madeira, Portugal, January 27–29, 2018, Revised Selected Papers (pp. 369-389). Springer Publishing Company. Communications in Computer and Information Science, Vol.. CCIS, volume 997 https://doi.org/10.1007/978-3-030-26756-8_18