Abstract

This article presents the issues related to applying computer
vision techniques to identify facial expressions and recognize the mood
of Traumatic Brain Injured (TBI) patients in real life scenarios. Many AQ1
TBI patients face serious problems in communication and activities of
daily living. These are due to restricted movement of muscles or paraly- AQ2
sis with lesser facial expression along with non-cooperative behaviour,
and inappropriate reasoning and reactions. All these aforementioned
attributes contribute towards the complexity of the system for the automatic understanding of their emotional expressions. Existing systems for
facial expression recognition are highly accurate when tested on healthy
people in controlled conditions. However, their performance is not yet
verified on the TBI patients in the real environment. In order to test
this, we devised a special arrangement to collect data from these patients.
Unlike the controlled environment, it was very challenging because these
patients have large pose variations, poor attention and concentration
with impulsive behaviours. In order to acquire high-quality facial images
from videos for facial expression analysis, effective techniques of data
preprocessing are applied. The extracted images are then fed to a deep
learning architecture based on Convolution Neural Network (CNN) and
Long Short-Term Memory (LSTM) network to exploit the spatiotemporal information with 3D face frontalization. RGB and thermal imaging
modalities are used and the experimental results show that better quality of facial images and larger database enhance the system performance
in facial expressions and mood recognition of TBI patients under natural challenging conditions. The proposed approach hopefully facilitates
the physiotherapists, trainers and caregivers to deploy fast rehabilitation
activities by knowing the positive mood of the patients.
Original languageEnglish
Title of host publicationSpringer Nature Switzerland AG 2019 : D. Bechmann et al. (Eds.): VISIGRAPP 2018, CCIS 997, pp. 1–21, 2019.
Number of pages21
Place of PublicationSpringer Nature Switzerland AG 2019
DOIs
Publication statusAccepted/In press - 2 Jul 2019

Fingerprint

Patient rehabilitation
Brain
Imaging techniques
Convolution
Muscle
Neural networks
Data storage equipment
Communication
Hot Temperature

Keywords

  • Computer vision
  • Multi-visual (RGB, thermal) modalities
  • Face detection
  • Facial Landmarks
  • Facial Expressions Recognition
  • Convolution Neural Networks
  • Long-Short Term Memory
  • Traumatic Brain Injured Patients

Cite this

Ilyas, C. M. A., Haque, M. A., Nasrollahi, K., Rehm, M., & Moeslund, T. B. (Accepted/In press). Effective Facial Expression Recognition Through Multimodal Imaging for Traumatic Brain Injured Patient’s Rehabilitation. In Springer Nature Switzerland AG 2019: D. Bechmann et al. (Eds.): VISIGRAPP 2018, CCIS 997, pp. 1–21, 2019. Springer Nature Switzerland AG 2019. https://doi.org/10.1007/978-3-030-26756-8_18
@inbook{ea2944e5550940d1b35dcf7f70ff2e98,
title = "Effective Facial Expression Recognition Through Multimodal Imaging for Traumatic Brain Injured Patient’s Rehabilitation",
abstract = "This article presents the issues related to applying computervision techniques to identify facial expressions and recognize the moodof Traumatic Brain Injured (TBI) patients in real life scenarios. Many AQ1TBI patients face serious problems in communication and activities ofdaily living. These are due to restricted movement of muscles or paraly- AQ2sis with lesser facial expression along with non-cooperative behaviour,and inappropriate reasoning and reactions. All these aforementionedattributes contribute towards the complexity of the system for the automatic understanding of their emotional expressions. Existing systems forfacial expression recognition are highly accurate when tested on healthypeople in controlled conditions. However, their performance is not yetverified on the TBI patients in the real environment. In order to testthis, we devised a special arrangement to collect data from these patients.Unlike the controlled environment, it was very challenging because thesepatients have large pose variations, poor attention and concentrationwith impulsive behaviours. In order to acquire high-quality facial imagesfrom videos for facial expression analysis, effective techniques of datapreprocessing are applied. The extracted images are then fed to a deeplearning architecture based on Convolution Neural Network (CNN) andLong Short-Term Memory (LSTM) network to exploit the spatiotemporal information with 3D face frontalization. RGB and thermal imagingmodalities are used and the experimental results show that better quality of facial images and larger database enhance the system performancein facial expressions and mood recognition of TBI patients under natural challenging conditions. The proposed approach hopefully facilitatesthe physiotherapists, trainers and caregivers to deploy fast rehabilitationactivities by knowing the positive mood of the patients.",
keywords = "Computer vision, Multi-visual (RGB, thermal) modalities, Face detection, Facial Landmarks, Facial Expressions Recognition, Convolution Neural Networks, Long-Short Term Memory, Traumatic Brain Injured Patients",
author = "Ilyas, {Chaudhary Muhammad Aqdus} and Haque, {Mohammad Ahsanul} and Kamal Nasrollahi and Matthias Rehm and Moeslund, {Thomas B.}",
year = "2019",
month = "7",
day = "2",
doi = "10.1007/978-3-030-26756-8_18",
language = "English",
booktitle = "Springer Nature Switzerland AG 2019",

}

Ilyas, CMA, Haque, MA, Nasrollahi, K, Rehm, M & Moeslund, TB 2019, Effective Facial Expression Recognition Through Multimodal Imaging for Traumatic Brain Injured Patient’s Rehabilitation. in Springer Nature Switzerland AG 2019: D. Bechmann et al. (Eds.): VISIGRAPP 2018, CCIS 997, pp. 1–21, 2019.. Springer Nature Switzerland AG 2019. https://doi.org/10.1007/978-3-030-26756-8_18

Effective Facial Expression Recognition Through Multimodal Imaging for Traumatic Brain Injured Patient’s Rehabilitation. / Ilyas, Chaudhary Muhammad Aqdus; Haque, Mohammad Ahsanul; Nasrollahi, Kamal; Rehm, Matthias; Moeslund, Thomas B.

Springer Nature Switzerland AG 2019: D. Bechmann et al. (Eds.): VISIGRAPP 2018, CCIS 997, pp. 1–21, 2019.. Springer Nature Switzerland AG 2019, 2019.

Research output: Contribution to book/anthology/report/conference proceedingBook chapterResearchpeer-review

TY - CHAP

T1 - Effective Facial Expression Recognition Through Multimodal Imaging for Traumatic Brain Injured Patient’s Rehabilitation

AU - Ilyas, Chaudhary Muhammad Aqdus

AU - Haque, Mohammad Ahsanul

AU - Nasrollahi, Kamal

AU - Rehm, Matthias

AU - Moeslund, Thomas B.

PY - 2019/7/2

Y1 - 2019/7/2

N2 - This article presents the issues related to applying computervision techniques to identify facial expressions and recognize the moodof Traumatic Brain Injured (TBI) patients in real life scenarios. Many AQ1TBI patients face serious problems in communication and activities ofdaily living. These are due to restricted movement of muscles or paraly- AQ2sis with lesser facial expression along with non-cooperative behaviour,and inappropriate reasoning and reactions. All these aforementionedattributes contribute towards the complexity of the system for the automatic understanding of their emotional expressions. Existing systems forfacial expression recognition are highly accurate when tested on healthypeople in controlled conditions. However, their performance is not yetverified on the TBI patients in the real environment. In order to testthis, we devised a special arrangement to collect data from these patients.Unlike the controlled environment, it was very challenging because thesepatients have large pose variations, poor attention and concentrationwith impulsive behaviours. In order to acquire high-quality facial imagesfrom videos for facial expression analysis, effective techniques of datapreprocessing are applied. The extracted images are then fed to a deeplearning architecture based on Convolution Neural Network (CNN) andLong Short-Term Memory (LSTM) network to exploit the spatiotemporal information with 3D face frontalization. RGB and thermal imagingmodalities are used and the experimental results show that better quality of facial images and larger database enhance the system performancein facial expressions and mood recognition of TBI patients under natural challenging conditions. The proposed approach hopefully facilitatesthe physiotherapists, trainers and caregivers to deploy fast rehabilitationactivities by knowing the positive mood of the patients.

AB - This article presents the issues related to applying computervision techniques to identify facial expressions and recognize the moodof Traumatic Brain Injured (TBI) patients in real life scenarios. Many AQ1TBI patients face serious problems in communication and activities ofdaily living. These are due to restricted movement of muscles or paraly- AQ2sis with lesser facial expression along with non-cooperative behaviour,and inappropriate reasoning and reactions. All these aforementionedattributes contribute towards the complexity of the system for the automatic understanding of their emotional expressions. Existing systems forfacial expression recognition are highly accurate when tested on healthypeople in controlled conditions. However, their performance is not yetverified on the TBI patients in the real environment. In order to testthis, we devised a special arrangement to collect data from these patients.Unlike the controlled environment, it was very challenging because thesepatients have large pose variations, poor attention and concentrationwith impulsive behaviours. In order to acquire high-quality facial imagesfrom videos for facial expression analysis, effective techniques of datapreprocessing are applied. The extracted images are then fed to a deeplearning architecture based on Convolution Neural Network (CNN) andLong Short-Term Memory (LSTM) network to exploit the spatiotemporal information with 3D face frontalization. RGB and thermal imagingmodalities are used and the experimental results show that better quality of facial images and larger database enhance the system performancein facial expressions and mood recognition of TBI patients under natural challenging conditions. The proposed approach hopefully facilitatesthe physiotherapists, trainers and caregivers to deploy fast rehabilitationactivities by knowing the positive mood of the patients.

KW - Computer vision

KW - Multi-visual (RGB, thermal) modalities

KW - Face detection

KW - Facial Landmarks

KW - Facial Expressions Recognition

KW - Convolution Neural Networks

KW - Long-Short Term Memory

KW - Traumatic Brain Injured Patients

U2 - 10.1007/978-3-030-26756-8_18

DO - 10.1007/978-3-030-26756-8_18

M3 - Book chapter

BT - Springer Nature Switzerland AG 2019

CY - Springer Nature Switzerland AG 2019

ER -

Ilyas CMA, Haque MA, Nasrollahi K, Rehm M, Moeslund TB. Effective Facial Expression Recognition Through Multimodal Imaging for Traumatic Brain Injured Patient’s Rehabilitation. In Springer Nature Switzerland AG 2019: D. Bechmann et al. (Eds.): VISIGRAPP 2018, CCIS 997, pp. 1–21, 2019.. Springer Nature Switzerland AG 2019. 2019 https://doi.org/10.1007/978-3-030-26756-8_18