Changes in Facial Expression as Biometric: A Database and Benchmarks of Identification

Rain Eric Haamer, Kaustubh Kulkarni, Nasrin Imanpour, Mohammad Ahsanul Haque, Egils Avots, Michelle Breisch, Kamal Nasrollahi, Sergio Escalera Guerrero, Cagri Ozcinar, Xavier Baro, Ahmad Reza Naghsh-Nilchi, Thomas B. Moeslund, Gholamreza Anbarjafari

Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

5 Citations (Scopus)
109 Downloads (Pure)

Abstract

Facial dynamics can be considered as unique signatures for discrimination between people. These have started to become important topic since many devices have the possibility of unlocking using face recognition or verification. In this work, we evaluate the efficacy of the transition frames of video in emotion as compared to the peak emotion frames for identification. For experiments with transition frames we extract features from each frame of the video from a fine-tuned VGG-Face Convolutional Neural Network (CNN) and geometric features from facial landmark points. To model the temporal context of the transition frames we train a Long-Short Term Memory (LSTM) on the geometric and the CNN features. Furthermore, we employ two fusion strategies: first, an early fusion, in which the geometric and the CNN features are stacked and fed to the LSTM. Second, a late fusion, in which the prediction of the LSTMs, trained
independently on the two features, are stacked and used with a Support Vector Machine (SVM). Experimental results show that the late fusion strategy gives the best results and the transition frames give better identification results as compared to the peak emotion frames.
Original languageEnglish
Title of host publication13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)
Number of pages8
PublisherIEEE
Publication date2018
Pages621-628
Article number8373891
ISBN (Print)978-1-5386-2336-7
ISBN (Electronic)978-1-5386-2335-0
DOIs
Publication statusPublished - 2018
EventIEEE Conf. on Automatic Face and Gesture Recognition Workshops - X'ian, China
Duration: 15 May 201819 May 2018
https://fg2018.cse.sc.edu

Conference

ConferenceIEEE Conf. on Automatic Face and Gesture Recognition Workshops
CountryChina
CityX'ian
Period15/05/201819/05/2018
Internet address

Fingerprint

Biometrics
Fusion reactions
Neural networks
Face recognition
Support vector machines
Experiments
Long short-term memory

Keywords

  • Facial expression
  • biometric
  • database
  • benchmark
  • Deep Learning
  • CNN
  • LSTM
  • Multimodal
  • Spatio-temporal
  • SVM

Cite this

Haamer, R. E., Kulkarni, K., Imanpour, N., Haque, M. A., Avots, E., Breisch, M., ... Anbarjafari, G. (2018). Changes in Facial Expression as Biometric: A Database and Benchmarks of Identification. In 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) (pp. 621-628). [8373891] IEEE. https://doi.org/10.1109/FG.2018.00098
Haamer, Rain Eric ; Kulkarni, Kaustubh ; Imanpour, Nasrin ; Haque, Mohammad Ahsanul ; Avots, Egils ; Breisch, Michelle ; Nasrollahi, Kamal ; Guerrero, Sergio Escalera ; Ozcinar, Cagri ; Baro, Xavier ; Naghsh-Nilchi, Ahmad Reza ; Moeslund, Thomas B. ; Anbarjafari, Gholamreza . / Changes in Facial Expression as Biometric : A Database and Benchmarks of Identification. 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018. pp. 621-628
@inproceedings{a81113492dac49ef85f80d0158a816bd,
title = "Changes in Facial Expression as Biometric: A Database and Benchmarks of Identification",
abstract = "Facial dynamics can be considered as unique signatures for discrimination between people. These have started to become important topic since many devices have the possibility of unlocking using face recognition or verification. In this work, we evaluate the efficacy of the transition frames of video in emotion as compared to the peak emotion frames for identification. For experiments with transition frames we extract features from each frame of the video from a fine-tuned VGG-Face Convolutional Neural Network (CNN) and geometric features from facial landmark points. To model the temporal context of the transition frames we train a Long-Short Term Memory (LSTM) on the geometric and the CNN features. Furthermore, we employ two fusion strategies: first, an early fusion, in which the geometric and the CNN features are stacked and fed to the LSTM. Second, a late fusion, in which the prediction of the LSTMs, trainedindependently on the two features, are stacked and used with a Support Vector Machine (SVM). Experimental results show that the late fusion strategy gives the best results and the transition frames give better identification results as compared to the peak emotion frames.",
keywords = "Facial expression, biometric, database, benchmark, Deep Learning, CNN, LSTM, Multimodal, Spatio-temporal, SVM",
author = "Haamer, {Rain Eric} and Kaustubh Kulkarni and Nasrin Imanpour and Haque, {Mohammad Ahsanul} and Egils Avots and Michelle Breisch and Kamal Nasrollahi and Guerrero, {Sergio Escalera} and Cagri Ozcinar and Xavier Baro and Naghsh-Nilchi, {Ahmad Reza} and Moeslund, {Thomas B.} and Gholamreza Anbarjafari",
year = "2018",
doi = "10.1109/FG.2018.00098",
language = "English",
isbn = "978-1-5386-2336-7",
pages = "621--628",
booktitle = "13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)",
publisher = "IEEE",
address = "United States",

}

Haamer, RE, Kulkarni, K, Imanpour, N, Haque, MA, Avots, E, Breisch, M, Nasrollahi, K, Guerrero, SE, Ozcinar, C, Baro, X, Naghsh-Nilchi, AR, Moeslund, TB & Anbarjafari, G 2018, Changes in Facial Expression as Biometric: A Database and Benchmarks of Identification. in 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)., 8373891, IEEE, pp. 621-628, IEEE Conf. on Automatic Face and Gesture Recognition Workshops, X'ian, China, 15/05/2018. https://doi.org/10.1109/FG.2018.00098

Changes in Facial Expression as Biometric : A Database and Benchmarks of Identification. / Haamer, Rain Eric; Kulkarni, Kaustubh; Imanpour, Nasrin; Haque, Mohammad Ahsanul; Avots, Egils; Breisch, Michelle; Nasrollahi, Kamal; Guerrero, Sergio Escalera; Ozcinar, Cagri; Baro, Xavier; Naghsh-Nilchi, Ahmad Reza; Moeslund, Thomas B.; Anbarjafari, Gholamreza .

13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018. p. 621-628 8373891.

Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

TY - GEN

T1 - Changes in Facial Expression as Biometric

T2 - A Database and Benchmarks of Identification

AU - Haamer, Rain Eric

AU - Kulkarni, Kaustubh

AU - Imanpour, Nasrin

AU - Haque, Mohammad Ahsanul

AU - Avots, Egils

AU - Breisch, Michelle

AU - Nasrollahi, Kamal

AU - Guerrero, Sergio Escalera

AU - Ozcinar, Cagri

AU - Baro, Xavier

AU - Naghsh-Nilchi, Ahmad Reza

AU - Moeslund, Thomas B.

AU - Anbarjafari, Gholamreza

PY - 2018

Y1 - 2018

N2 - Facial dynamics can be considered as unique signatures for discrimination between people. These have started to become important topic since many devices have the possibility of unlocking using face recognition or verification. In this work, we evaluate the efficacy of the transition frames of video in emotion as compared to the peak emotion frames for identification. For experiments with transition frames we extract features from each frame of the video from a fine-tuned VGG-Face Convolutional Neural Network (CNN) and geometric features from facial landmark points. To model the temporal context of the transition frames we train a Long-Short Term Memory (LSTM) on the geometric and the CNN features. Furthermore, we employ two fusion strategies: first, an early fusion, in which the geometric and the CNN features are stacked and fed to the LSTM. Second, a late fusion, in which the prediction of the LSTMs, trainedindependently on the two features, are stacked and used with a Support Vector Machine (SVM). Experimental results show that the late fusion strategy gives the best results and the transition frames give better identification results as compared to the peak emotion frames.

AB - Facial dynamics can be considered as unique signatures for discrimination between people. These have started to become important topic since many devices have the possibility of unlocking using face recognition or verification. In this work, we evaluate the efficacy of the transition frames of video in emotion as compared to the peak emotion frames for identification. For experiments with transition frames we extract features from each frame of the video from a fine-tuned VGG-Face Convolutional Neural Network (CNN) and geometric features from facial landmark points. To model the temporal context of the transition frames we train a Long-Short Term Memory (LSTM) on the geometric and the CNN features. Furthermore, we employ two fusion strategies: first, an early fusion, in which the geometric and the CNN features are stacked and fed to the LSTM. Second, a late fusion, in which the prediction of the LSTMs, trainedindependently on the two features, are stacked and used with a Support Vector Machine (SVM). Experimental results show that the late fusion strategy gives the best results and the transition frames give better identification results as compared to the peak emotion frames.

KW - Facial expression

KW - biometric

KW - database

KW - benchmark

KW - Deep Learning

KW - CNN

KW - LSTM

KW - Multimodal

KW - Spatio-temporal

KW - SVM

U2 - 10.1109/FG.2018.00098

DO - 10.1109/FG.2018.00098

M3 - Article in proceeding

SN - 978-1-5386-2336-7

SP - 621

EP - 628

BT - 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)

PB - IEEE

ER -

Haamer RE, Kulkarni K, Imanpour N, Haque MA, Avots E, Breisch M et al. Changes in Facial Expression as Biometric: A Database and Benchmarks of Identification. In 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE. 2018. p. 621-628. 8373891 https://doi.org/10.1109/FG.2018.00098