Multimodal Heartbeat Rate Estimation from the Fusion of Facial RGB and Thermal Videos

Anders Skaarup Johansen, Jesper Wædeled Henriksen, Mohammad Ahsanul Haque, Mohammad Naser Sabet Jahromi, Kamal Nasrollahi, Thomas B. Moeslund

Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

227 Downloads (Pure)


Measuring Heartbeat Rate (HR) is an important tool for monitoring the health of a person. When the heart beats the influx of blood to the head causes slight involuntary movement and subtle skin color changes, which cannot be seen by the naked eye but can be tracked from facial videos using computer vision techniques and can be analyzed to estimate the HR. However, the current state of the art solutions encounter an increasing amount of complications when the subject has voluntary motion on the face or when the lighting conditions change in the video. Thus the accuracy of the HR estimation using computer vision is still inferior to that of a physical Electrocardiography (ECG) based system. The aim of this work is to improve the current non-invasive HR measurement by fusing the motion-based and color-based HR estimation methods and using them on multiple input modalities, e.g., RGB and thermal imaging. Our experiments indicate that late-fusion of the results of these methods (motion and color-based) applied to these different modalities, produces more accurate results compared to the existing solutions.
Original languageEnglish
Title of host publicationEleventh International Conference on Machine Vision, ICMV 2018
EditorsJianhong Zhou, Antanas Verikas, Dmitry P. Nikolaev, Petia Radeva
Number of pages8
PublisherSPIE - International Society for Optical Engineering
Publication date2019
Article number110410R
ISBN (Print)9781510627482
ISBN (Electronic)9781510627482
Publication statusPublished - 2019
EventThe 11th International Conference on Machine Vision - Munich, Germany
Duration: 1 Nov 20183 Nov 2018


ConferenceThe 11th International Conference on Machine Vision
SeriesProceedings of SPIE, the International Society for Optical Engineering


  • Heartbeat
  • Heartbeat rate
  • Facial video
  • RGB
  • Video
  • Thermal
  • Multimodal
  • PPG
  • Camera

Fingerprint Dive into the research topics of 'Multimodal Heartbeat Rate Estimation from the Fusion of Facial RGB and Thermal Videos'. Together they form a unique fingerprint.

Cite this