Teaching Pepper Robot to Recognize Emotions of Traumatic Brain Injured Patients Using Deep Neural Networks

Chaudhary Muhammad Aqdus Ilyas; Viktor Schmuck; Mohammad Ahsanul Haque; Kamal Nasrollahi; Matthias Rehm; Thomas B. Moeslund

doi:10.1109/RO-MAN46459.2019.8956445

Teaching Pepper Robot to Recognize Emotions of Traumatic Brain Injured Patients Using Deep Neural Networks

Chaudhary Muhammad Aqdus Ilyas, Viktor Schmuck, Mohammad Ahsanul Haque, Kamal Nasrollahi, Matthias Rehm, Thomas B. Moeslund

Research output: Contribution to book/anthology/report/conference proceeding › Article in proceeding › Research › peer-review

7 Citations (Scopus)

209 Downloads (Pure)

Abstract

Social signal extraction from the facial analysis is
a popular research area in human-robot interaction. However,
recognition of emotional signals from Traumatic Brain Injured
(TBI) patients with the help of robots and non-intrusive sensors
is yet to be explored. Existing robots have limited abilities to
automatically identify human emotions and respond accordingly. Their interaction with TBI patients could be even more
challenging and complex due to unique, unusual and diverse
ways of expressing their emotions. To tackle the disparity
in a TBI patient’s Facial Expressions (FEs), a specialized
deep-trained model for automatic detection of TBI patients’
emotions and FE (TBI-FER model) is designed, for robotassisted rehabilitation activities. In addition, the Pepper robot’s
built-in model for FE is investigated on TBI patients as well
as on healthy people. Variance in their emotional expressions
is determined by comparative studies. It is observed that the
customized trained system is highly essential for the deployment
of Pepper robot as a Socially Assistive Robot (SAR).

Original language	English
Title of host publication	28th IEEE International Conference on Robot and Human Interactive Communication (ROMAN)
Publisher	IEEE
Publication date	12 Oct 2019
Article number	8956445
ISBN (Electronic)	978-1-7281-2622-7
DOIs	https://doi.org/10.1109/RO-MAN46459.2019.8956445
Publication status	Published - 12 Oct 2019
Event	The 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN): Ro-man2019 - New Delhi, India Duration: 14 Oct 2019 → 18 Nov 2019 https://ro-man2019.org/

Conference

Conference	The 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)
Country/Territory	India
City	New Delhi
Period	14/10/2019 → 18/11/2019
Internet address	https://ro-man2019.org/

Series	IEEE RO-MAN proceedings
ISSN	1944-9445

Keywords

Robots in Education, Therapy and Rehabilitation, Non-verbal Cues and Expressiveness, Applications of Social Robots

Access to Document

10.1109/RO-MAN46459.2019.8956445

Teaching_Pepper_Robot_to_Recognize_Emotions_of_Traumatic_Brain_Injured__Patients_Using_Deep_Neural_Networks_ArchivedSubmitted manuscript, 5.2 MB

AUB Link

Search for the material in Aalborg University Library's search engine

1 Finished

Handling stress reactions in institutional care by adapting interaction behavior of a social robot
Ilyas, C. M., Nasrollahi, K., Rehm, M. & Moeslund, T. B.
03/04/2017 → 14/04/2020
Project: Research

7 Citations
1 PhD thesis

Facial Emotion Recognition for Citizens with Traumatic Brain Injury for Therapeutic Robot Interaction
Ilyas, C. M., 2021, Aalborg Universitetsforlag. 196 p.
Research output: PhD thesis

Open Access
File

151 Downloads (Pure)

Cite this

@inproceedings{f25ce61410e74118a6a49859556b3496,

title = "Teaching Pepper Robot to Recognize Emotions of Traumatic Brain Injured Patients Using Deep Neural Networks",

abstract = "Social signal extraction from the facial analysis isa popular research area in human-robot interaction. However,recognition of emotional signals from Traumatic Brain Injured(TBI) patients with the help of robots and non-intrusive sensorsis yet to be explored. Existing robots have limited abilities toautomatically identify human emotions and respond accordingly. Their interaction with TBI patients could be even morechallenging and complex due to unique, unusual and diverseways of expressing their emotions. To tackle the disparityin a TBI patient{\textquoteright}s Facial Expressions (FEs), a specializeddeep-trained model for automatic detection of TBI patients{\textquoteright}emotions and FE (TBI-FER model) is designed, for robotassisted rehabilitation activities. In addition, the Pepper robot{\textquoteright}sbuilt-in model for FE is investigated on TBI patients as wellas on healthy people. Variance in their emotional expressionsis determined by comparative studies. It is observed that thecustomized trained system is highly essential for the deploymentof Pepper robot as a Socially Assistive Robot (SAR).",

keywords = "Robots in Education, Therapy and Rehabilitation, Non-verbal Cues and Expressiveness, Applications of Social Robots",

author = "Ilyas, {Chaudhary Muhammad Aqdus} and Viktor Schmuck and Haque, {Mohammad Ahsanul} and Kamal Nasrollahi and Matthias Rehm and Moeslund, {Thomas B.}",

year = "2019",

month = oct,

day = "12",

doi = "10.1109/RO-MAN46459.2019.8956445",

language = "English",

series = "IEEE RO-MAN proceedings",

publisher = "IEEE",

booktitle = "28th IEEE International Conference on Robot and Human Interactive Communication (ROMAN)",

address = "United States",

note = "The 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN) : Ro-man2019 ; Conference date: 14-10-2019 Through 18-11-2019",

url = "https://ro-man2019.org/",

}

Ilyas, CMA, Schmuck, V, Haque, MA, Nasrollahi, K , Rehm, M & Moeslund, TB 2019, Teaching Pepper Robot to Recognize Emotions of Traumatic Brain Injured Patients Using Deep Neural Networks. in 28th IEEE International Conference on Robot and Human Interactive Communication (ROMAN)., 8956445, IEEE, IEEE RO-MAN proceedings, The 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), New Delhi, India, 14/10/2019. https://doi.org/10.1109/RO-MAN46459.2019.8956445

Teaching Pepper Robot to Recognize Emotions of Traumatic Brain Injured Patients Using Deep Neural Networks. / Ilyas, Chaudhary Muhammad Aqdus; Schmuck, Viktor; Haque, Mohammad Ahsanul et al.
28th IEEE International Conference on Robot and Human Interactive Communication (ROMAN). IEEE, 2019. 8956445 (IEEE RO-MAN proceedings).

Research output: Contribution to book/anthology/report/conference proceeding › Article in proceeding › Research › peer-review

TY - GEN

T1 - Teaching Pepper Robot to Recognize Emotions of Traumatic Brain Injured Patients Using Deep Neural Networks

AU - Ilyas, Chaudhary Muhammad Aqdus

AU - Schmuck, Viktor

AU - Haque, Mohammad Ahsanul

AU - Nasrollahi, Kamal

AU - Rehm, Matthias

AU - Moeslund, Thomas B.

PY - 2019/10/12

Y1 - 2019/10/12

N2 - Social signal extraction from the facial analysis isa popular research area in human-robot interaction. However,recognition of emotional signals from Traumatic Brain Injured(TBI) patients with the help of robots and non-intrusive sensorsis yet to be explored. Existing robots have limited abilities toautomatically identify human emotions and respond accordingly. Their interaction with TBI patients could be even morechallenging and complex due to unique, unusual and diverseways of expressing their emotions. To tackle the disparityin a TBI patient’s Facial Expressions (FEs), a specializeddeep-trained model for automatic detection of TBI patients’emotions and FE (TBI-FER model) is designed, for robotassisted rehabilitation activities. In addition, the Pepper robot’sbuilt-in model for FE is investigated on TBI patients as wellas on healthy people. Variance in their emotional expressionsis determined by comparative studies. It is observed that thecustomized trained system is highly essential for the deploymentof Pepper robot as a Socially Assistive Robot (SAR).

AB - Social signal extraction from the facial analysis isa popular research area in human-robot interaction. However,recognition of emotional signals from Traumatic Brain Injured(TBI) patients with the help of robots and non-intrusive sensorsis yet to be explored. Existing robots have limited abilities toautomatically identify human emotions and respond accordingly. Their interaction with TBI patients could be even morechallenging and complex due to unique, unusual and diverseways of expressing their emotions. To tackle the disparityin a TBI patient’s Facial Expressions (FEs), a specializeddeep-trained model for automatic detection of TBI patients’emotions and FE (TBI-FER model) is designed, for robotassisted rehabilitation activities. In addition, the Pepper robot’sbuilt-in model for FE is investigated on TBI patients as wellas on healthy people. Variance in their emotional expressionsis determined by comparative studies. It is observed that thecustomized trained system is highly essential for the deploymentof Pepper robot as a Socially Assistive Robot (SAR).

KW - Robots in Education, Therapy and Rehabilitation, Non-verbal Cues and Expressiveness, Applications of Social Robots

U2 - 10.1109/RO-MAN46459.2019.8956445

DO - 10.1109/RO-MAN46459.2019.8956445

M3 - Article in proceeding

T3 - IEEE RO-MAN proceedings

BT - 28th IEEE International Conference on Robot and Human Interactive Communication (ROMAN)

PB - IEEE

T2 - The 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)

Y2 - 14 October 2019 through 18 November 2019

ER -

Teaching Pepper Robot to Recognize Emotions of Traumatic Brain Injured Patients Using Deep Neural Networks

Abstract

Conference

Keywords

Access to Document

AUB Link

Fingerprint

Projects

Handling stress reactions in institutional care by adapting interaction behavior of a social robot

Research output

Facial Emotion Recognition for Citizens with Traumatic Brain Injury for Therapeutic Robot Interaction

Cite this