Project Details


This project is related to applying computer vision techniques to identify facial expressions and recognize the mood of Traumatic Brain Injured (TBI) patients in real life scenarios. Many TBI patients face serious problems in communication and activities of daily living. These are due to restricted movement of muscles or paralysis with lesser facial expression along with non-cooperative behavior, and inappropriate reasoning and reactions. All these aforementioned attributes contribute towards the complexity of the system for the automatic understanding of their emotional expressions. Existing systems for facial expression recognition are highly accurate when tested on healthy people in controlled conditions. However, their performance is not yet verified on the TBI patients in the real environment. In order to test this, we devised a special arrangement to collect data from these patients. Unlike the controlled environment, it is very challenging because these patients have large pose variations, poor attention, and concentration with impulsive behaviors. In order to acquire high-quality facial images from videos for facial expression analysis, effective techniques of data preprocessing are applied. The extracted images are then fed to a deep learning architecture based on Convolution Neural Network (CNN) and Long Short-Term Memory (LSTM) network to exploit the spatiotemporal information with 3D face frontalization. RGB, thermal and depth imaging modalities are used and the results will be analyzed.

Key findings

TBI patients Database, Facial Expression Recognition, Mood Identification, Pain and Neutral Expressions for TBI patients
Short titleCVT for TBI
Effective start/end date03/04/201720/04/2020

Research Output

  • 2 Article in proceeding

Facial Expression Recognition for Traumatic Brain Injured Patients

Ilyas, C. M. A., Haque, M. A., Rehm, M., Nasrollahi, K. & Moeslund, T. B., 2018, VISAPP 2018: 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications.. Tremeau, A., Braz, J. & Imai, F. (eds.). SCITEPRESS Digital Library, Vol. 4. p. 522-530 9 p.

Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

Open Access
  • 4 Citations (Scopus)
    399 Downloads (Pure)

    Rehabilitation of Traumatic Brain Injured Patients: Patient Mood Analysis from Multimodal Video

    Ilyas, C. M. A., Nasrollahi, K., Rehm, M. & Moeslund, T. B., 7 Oct 2018, 2018 IEEE International Conference on Image Processing: ICIP 2018. IEEE, p. 2291-2295 5 p. 8451223. (IEEE International Conference on Image Processing (ICIP)).

    Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

  • 1 Citation (Scopus)
    68 Downloads (Pure)