135 Downloads (Pure)

Abstract

Measuring Heartbeat Rate (HR) is an important tool for monitoring the health of a person. When the heart beats the influx of blood to the head causes slight involuntary movement and subtle skin color changes, which cannot be seen by the naked eye but can be tracked from facial videos using computer vision techniques and can be analyzed to estimate the HR. However, the current state of the art solutions encounter an increasing amount of complications when the subject has voluntary motion on the face or when the lighting conditions change in the video. Thus the accuracy of the HR estimation using computer vision is still inferior to that of a physical Electrocardiography (ECG) based system. The aim of this work is to improve the current non-invasive HR measurement by fusing the motion-based and color-based HR estimation methods and using them on multiple input modalities, e.g., RGB and thermal imaging. Our experiments indicate that late-fusion of the results of these methods (motion and color-based) applied to these different modalities, produces more accurate results compared to the existing solutions.
Original languageEnglish
Title of host publication11th International Conference on Machine Vision (ICMV)
Number of pages8
Volume1104101
PublisherSPIE - International Society for Optical Engineering
Publication date2019
Pages110410R
Publication statusPublished - 2019
EventThe 11th International Conference on Machine Vision - Munich, Germany
Duration: 1 Nov 20183 Nov 2018

Conference

ConferenceThe 11th International Conference on Machine Vision
CountryGermany
CityMunich
Period01/11/201803/11/2018

Fingerprint

Fusion reactions
Color
Computer vision
Infrared imaging
Electrocardiography
Skin
Blood
Lighting
Health
Monitoring
Hot Temperature
Experiments

Keywords

  • Heartbeat
  • Heartbeat rate
  • Facial video
  • RGB
  • Video
  • Thermal
  • Multimodal
  • PPG
  • Camera

Cite this

Johansen, A. S., Henriksen, J. W., Haque, M. A., Jahromi, M. N. S., Nasrollahi, K., & Moeslund, T. B. (2019). Multimodal Heartbeat Rate Estimation from the Fusion of Facial RGB and Thermal Videos. In 11th International Conference on Machine Vision (ICMV) (Vol. 1104101, pp. 110410R ). SPIE - International Society for Optical Engineering.
Johansen, Anders Skaarup ; Henriksen, Jesper Wædeled ; Haque, Mohammad Ahsanul ; Jahromi, Mohammad Naser Sabet ; Nasrollahi, Kamal ; Moeslund, Thomas B. / Multimodal Heartbeat Rate Estimation from the Fusion of Facial RGB and Thermal Videos. 11th International Conference on Machine Vision (ICMV). Vol. 1104101 SPIE - International Society for Optical Engineering, 2019. pp. 110410R
@inproceedings{b64f33f5ab9d4d0d8470f21e0c2e634d,
title = "Multimodal Heartbeat Rate Estimation from the Fusion of Facial RGB and Thermal Videos",
abstract = "Measuring Heartbeat Rate (HR) is an important tool for monitoring the health of a person. When the heart beats the influx of blood to the head causes slight involuntary movement and subtle skin color changes, which cannot be seen by the naked eye but can be tracked from facial videos using computer vision techniques and can be analyzed to estimate the HR. However, the current state of the art solutions encounter an increasing amount of complications when the subject has voluntary motion on the face or when the lighting conditions change in the video. Thus the accuracy of the HR estimation using computer vision is still inferior to that of a physical Electrocardiography (ECG) based system. The aim of this work is to improve the current non-invasive HR measurement by fusing the motion-based and color-based HR estimation methods and using them on multiple input modalities, e.g., RGB and thermal imaging. Our experiments indicate that late-fusion of the results of these methods (motion and color-based) applied to these different modalities, produces more accurate results compared to the existing solutions.",
keywords = "Heartbeat, Heartbeat rate, Facial video, RGB, Video, Thermal, Multimodal, PPG, Camera",
author = "Johansen, {Anders Skaarup} and Henriksen, {Jesper W{\ae}deled} and Haque, {Mohammad Ahsanul} and Jahromi, {Mohammad Naser Sabet} and Kamal Nasrollahi and Moeslund, {Thomas B.}",
year = "2019",
language = "English",
volume = "1104101",
pages = "110410R",
booktitle = "11th International Conference on Machine Vision (ICMV)",
publisher = "SPIE - International Society for Optical Engineering",
address = "United States",

}

Johansen, AS, Henriksen, JW, Haque, MA, Jahromi, MNS, Nasrollahi, K & Moeslund, TB 2019, Multimodal Heartbeat Rate Estimation from the Fusion of Facial RGB and Thermal Videos. in 11th International Conference on Machine Vision (ICMV). vol. 1104101, SPIE - International Society for Optical Engineering, pp. 110410R , The 11th International Conference on Machine Vision, Munich, Germany, 01/11/2018.

Multimodal Heartbeat Rate Estimation from the Fusion of Facial RGB and Thermal Videos. / Johansen, Anders Skaarup ; Henriksen, Jesper Wædeled; Haque, Mohammad Ahsanul; Jahromi, Mohammad Naser Sabet; Nasrollahi, Kamal; Moeslund, Thomas B.

11th International Conference on Machine Vision (ICMV). Vol. 1104101 SPIE - International Society for Optical Engineering, 2019. p. 110410R .

Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

TY - GEN

T1 - Multimodal Heartbeat Rate Estimation from the Fusion of Facial RGB and Thermal Videos

AU - Johansen, Anders Skaarup

AU - Henriksen, Jesper Wædeled

AU - Haque, Mohammad Ahsanul

AU - Jahromi, Mohammad Naser Sabet

AU - Nasrollahi, Kamal

AU - Moeslund, Thomas B.

PY - 2019

Y1 - 2019

N2 - Measuring Heartbeat Rate (HR) is an important tool for monitoring the health of a person. When the heart beats the influx of blood to the head causes slight involuntary movement and subtle skin color changes, which cannot be seen by the naked eye but can be tracked from facial videos using computer vision techniques and can be analyzed to estimate the HR. However, the current state of the art solutions encounter an increasing amount of complications when the subject has voluntary motion on the face or when the lighting conditions change in the video. Thus the accuracy of the HR estimation using computer vision is still inferior to that of a physical Electrocardiography (ECG) based system. The aim of this work is to improve the current non-invasive HR measurement by fusing the motion-based and color-based HR estimation methods and using them on multiple input modalities, e.g., RGB and thermal imaging. Our experiments indicate that late-fusion of the results of these methods (motion and color-based) applied to these different modalities, produces more accurate results compared to the existing solutions.

AB - Measuring Heartbeat Rate (HR) is an important tool for monitoring the health of a person. When the heart beats the influx of blood to the head causes slight involuntary movement and subtle skin color changes, which cannot be seen by the naked eye but can be tracked from facial videos using computer vision techniques and can be analyzed to estimate the HR. However, the current state of the art solutions encounter an increasing amount of complications when the subject has voluntary motion on the face or when the lighting conditions change in the video. Thus the accuracy of the HR estimation using computer vision is still inferior to that of a physical Electrocardiography (ECG) based system. The aim of this work is to improve the current non-invasive HR measurement by fusing the motion-based and color-based HR estimation methods and using them on multiple input modalities, e.g., RGB and thermal imaging. Our experiments indicate that late-fusion of the results of these methods (motion and color-based) applied to these different modalities, produces more accurate results compared to the existing solutions.

KW - Heartbeat

KW - Heartbeat rate

KW - Facial video

KW - RGB

KW - Video

KW - Thermal

KW - Multimodal

KW - PPG

KW - Camera

M3 - Article in proceeding

VL - 1104101

SP - 110410R

BT - 11th International Conference on Machine Vision (ICMV)

PB - SPIE - International Society for Optical Engineering

ER -

Johansen AS, Henriksen JW, Haque MA, Jahromi MNS, Nasrollahi K, Moeslund TB. Multimodal Heartbeat Rate Estimation from the Fusion of Facial RGB and Thermal Videos. In 11th International Conference on Machine Vision (ICMV). Vol. 1104101. SPIE - International Society for Optical Engineering. 2019. p. 110410R