Deep Pain: Exploiting Long Short-Term Memory Networks for Facial Expression Classification

Pau Rodriguez, Guillem Cucurull, Jordi Gonzàlez, Josep M. Gonfaus, Kamal Nasrollahi, Thomas B. Moeslund, F. Xavier Roca

Research output: Contribution to journalJournal articleResearchpeer-review

23 Citations (Scopus)
711 Downloads (Pure)

Abstract

Pain is an unpleasant feeling that has been shown to be an important factor for the recovery of patients. Since this is costly in human resources and difficult to do objectively, there is the need for automatic systems to measure it. In this paper, con- trary to current state-of-the-art techniques in pain assessment, which are based on facial features only, we suggest that the performance can be enhanced by feeding the raw frames to deep learning models, outperforming the latest state-of-the-art results while also directly facing the problem of imbalanced data. As a baseline, our approach first uses convolutional neural networks (CNN) to learned facial features from VGG Faces, which are then linked to a Long Short-Term Memory (LSTM) to exploit the temporal relation between video frames. We further compare the performances of using the so popular schema based on the canonically normalized appearance versus taking into account the whole image: As a result, we outperform current state- of-the-art AUC performance in the UNBC-McMaster Shoulder Pain Expression Archive Database. In addition, to evaluate the generalization properties of our proposed methodology on facial motion recognition, we also report competitive results in the Cohn Kanade+ facial expression database.
Original languageEnglish
JournalI E E E Transactions on Cybernetics
VolumePP
Issue number99
Pages (from-to)1-11
ISSN2168-2267
DOIs
Publication statusE-pub ahead of print - 2017

Fingerprint

Personnel
Neural networks
Recovery
Long short-term memory
Deep learning

Keywords

  • deep learning
  • pain recognition

Cite this

Rodriguez, Pau ; Cucurull, Guillem ; Gonzàlez, Jordi ; M. Gonfaus, Josep ; Nasrollahi, Kamal ; Moeslund, Thomas B. ; Xavier Roca, F. / Deep Pain : Exploiting Long Short-Term Memory Networks for Facial Expression Classification. In: I E E E Transactions on Cybernetics. 2017 ; Vol. PP, No. 99. pp. 1-11.
@article{56823ffc8ae0420fb235f64833bd72cf,
title = "Deep Pain: Exploiting Long Short-Term Memory Networks for Facial Expression Classification",
abstract = "Pain is an unpleasant feeling that has been shown to be an important factor for the recovery of patients. Since this is costly in human resources and difficult to do objectively, there is the need for automatic systems to measure it. In this paper, con- trary to current state-of-the-art techniques in pain assessment, which are based on facial features only, we suggest that the performance can be enhanced by feeding the raw frames to deep learning models, outperforming the latest state-of-the-art results while also directly facing the problem of imbalanced data. As a baseline, our approach first uses convolutional neural networks (CNN) to learned facial features from VGG Faces, which are then linked to a Long Short-Term Memory (LSTM) to exploit the temporal relation between video frames. We further compare the performances of using the so popular schema based on the canonically normalized appearance versus taking into account the whole image: As a result, we outperform current state- of-the-art AUC performance in the UNBC-McMaster Shoulder Pain Expression Archive Database. In addition, to evaluate the generalization properties of our proposed methodology on facial motion recognition, we also report competitive results in the Cohn Kanade+ facial expression database.",
keywords = "deep learning, pain recognition",
author = "Pau Rodriguez and Guillem Cucurull and Jordi Gonz{\`a}lez and {M. Gonfaus}, Josep and Kamal Nasrollahi and Moeslund, {Thomas B.} and {Xavier Roca}, F.",
year = "2017",
doi = "10.1109/TCYB.2017.2662199",
language = "English",
volume = "PP",
pages = "1--11",
journal = "I E E E Transactions on Systems, Man and Cybernetics, Part B: Cybernetics",
issn = "1083-4419",
publisher = "IEEE",
number = "99",

}

Deep Pain : Exploiting Long Short-Term Memory Networks for Facial Expression Classification. / Rodriguez, Pau; Cucurull, Guillem; Gonzàlez, Jordi; M. Gonfaus, Josep ; Nasrollahi, Kamal; Moeslund, Thomas B.; Xavier Roca, F.

In: I E E E Transactions on Cybernetics, Vol. PP, No. 99, 2017, p. 1-11.

Research output: Contribution to journalJournal articleResearchpeer-review

TY - JOUR

T1 - Deep Pain

T2 - Exploiting Long Short-Term Memory Networks for Facial Expression Classification

AU - Rodriguez, Pau

AU - Cucurull, Guillem

AU - Gonzàlez, Jordi

AU - M. Gonfaus, Josep

AU - Nasrollahi, Kamal

AU - Moeslund, Thomas B.

AU - Xavier Roca, F.

PY - 2017

Y1 - 2017

N2 - Pain is an unpleasant feeling that has been shown to be an important factor for the recovery of patients. Since this is costly in human resources and difficult to do objectively, there is the need for automatic systems to measure it. In this paper, con- trary to current state-of-the-art techniques in pain assessment, which are based on facial features only, we suggest that the performance can be enhanced by feeding the raw frames to deep learning models, outperforming the latest state-of-the-art results while also directly facing the problem of imbalanced data. As a baseline, our approach first uses convolutional neural networks (CNN) to learned facial features from VGG Faces, which are then linked to a Long Short-Term Memory (LSTM) to exploit the temporal relation between video frames. We further compare the performances of using the so popular schema based on the canonically normalized appearance versus taking into account the whole image: As a result, we outperform current state- of-the-art AUC performance in the UNBC-McMaster Shoulder Pain Expression Archive Database. In addition, to evaluate the generalization properties of our proposed methodology on facial motion recognition, we also report competitive results in the Cohn Kanade+ facial expression database.

AB - Pain is an unpleasant feeling that has been shown to be an important factor for the recovery of patients. Since this is costly in human resources and difficult to do objectively, there is the need for automatic systems to measure it. In this paper, con- trary to current state-of-the-art techniques in pain assessment, which are based on facial features only, we suggest that the performance can be enhanced by feeding the raw frames to deep learning models, outperforming the latest state-of-the-art results while also directly facing the problem of imbalanced data. As a baseline, our approach first uses convolutional neural networks (CNN) to learned facial features from VGG Faces, which are then linked to a Long Short-Term Memory (LSTM) to exploit the temporal relation between video frames. We further compare the performances of using the so popular schema based on the canonically normalized appearance versus taking into account the whole image: As a result, we outperform current state- of-the-art AUC performance in the UNBC-McMaster Shoulder Pain Expression Archive Database. In addition, to evaluate the generalization properties of our proposed methodology on facial motion recognition, we also report competitive results in the Cohn Kanade+ facial expression database.

KW - deep learning

KW - pain recognition

UR - http://www.scopus.com/inward/record.url?scp=85012977472&partnerID=8YFLogxK

U2 - 10.1109/TCYB.2017.2662199

DO - 10.1109/TCYB.2017.2662199

M3 - Journal article

VL - PP

SP - 1

EP - 11

JO - I E E E Transactions on Systems, Man and Cybernetics, Part B: Cybernetics

JF - I E E E Transactions on Systems, Man and Cybernetics, Part B: Cybernetics

SN - 1083-4419

IS - 99

ER -