Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems

Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

Abstract

Humans tend to change their way of speaking when they are immersed in a noisy environment, a reflex known as Lombard effect. Current speech enhancement systems based on deep learning do not usually take into account this change in the speaking style, because they are trained with neutral (non-Lombard) speech utterances recorded under quiet conditions to which noise is artificially added. In this paper, we investigate the effects that the Lombard reflex has on the performance of audio-visual speech enhancement systems based on deep learning. The results show that a gap in the performance of as much as approximately 5 dB between the systems trained on neutral speech and the ones trained on Lombard speech exists. This indicates the benefit of taking into account the mismatch between neutral and Lombard speech in the design of audio-visual speech enhancement systems.

Original languageEnglish
Title of host publicationICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Number of pages5
PublisherIEEE
Publication date17 Apr 2019
Pages6615-6619
Article number8682713
ISBN (Print)978-1-4799-8132-8
ISBN (Electronic)978-1-4799-8131-1
DOIs
Publication statusPublished - 17 Apr 2019
Event2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) - Brighton, United Kingdom
Duration: 12 May 201917 May 2019
https://2019.ieeeicassp.org/

Conference

Conference2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
CountryUnited Kingdom
CityBrighton
Period12/05/201917/05/2019
Internet address
SeriesI E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings
ISSN1520-6149

Fingerprint

Speech enhancement
Deep learning

Keywords

  • Audio-visual speech enhancement
  • Lombard effect
  • deep learning

Cite this

Michelsanti, D., Tan, Z-H., Sigurdsson, S., & Jensen, J. (2019). Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6615-6619). [8682713] IEEE. I E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings https://doi.org/10.1109/ICASSP.2019.8682713
Michelsanti, Daniel ; Tan, Zheng-Hua ; Sigurdsson, Sigurdur ; Jensen, Jesper. / Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems. ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019. pp. 6615-6619 (I E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings).
@inproceedings{17da3cfcd9684373acc52ccbe20201fc,
title = "Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems",
abstract = "Humans tend to change their way of speaking when they are immersed in a noisy environment, a reflex known as Lombard effect. Current speech enhancement systems based on deep learning do not usually take into account this change in the speaking style, because they are trained with neutral (non-Lombard) speech utterances recorded under quiet conditions to which noise is artificially added. In this paper, we investigate the effects that the Lombard reflex has on the performance of audio-visual speech enhancement systems based on deep learning. The results show that a gap in the performance of as much as approximately 5 dB between the systems trained on neutral speech and the ones trained on Lombard speech exists. This indicates the benefit of taking into account the mismatch between neutral and Lombard speech in the design of audio-visual speech enhancement systems.",
keywords = "Audio-visual speech enhancement, Lombard effect, deep learning",
author = "Daniel Michelsanti and Zheng-Hua Tan and Sigurdur Sigurdsson and Jesper Jensen",
year = "2019",
month = "4",
day = "17",
doi = "10.1109/ICASSP.2019.8682713",
language = "English",
isbn = "978-1-4799-8132-8",
pages = "6615--6619",
booktitle = "ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)",
publisher = "IEEE",
address = "United States",

}

Michelsanti, D, Tan, Z-H, Sigurdsson, S & Jensen, J 2019, Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems. in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)., 8682713, IEEE, I E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings, pp. 6615-6619, 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, United Kingdom, 12/05/2019. https://doi.org/10.1109/ICASSP.2019.8682713

Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems. / Michelsanti, Daniel; Tan, Zheng-Hua; Sigurdsson, Sigurdur; Jensen, Jesper.

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019. p. 6615-6619 8682713 (I E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings).

Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

TY - GEN

T1 - Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems

AU - Michelsanti, Daniel

AU - Tan, Zheng-Hua

AU - Sigurdsson, Sigurdur

AU - Jensen, Jesper

PY - 2019/4/17

Y1 - 2019/4/17

N2 - Humans tend to change their way of speaking when they are immersed in a noisy environment, a reflex known as Lombard effect. Current speech enhancement systems based on deep learning do not usually take into account this change in the speaking style, because they are trained with neutral (non-Lombard) speech utterances recorded under quiet conditions to which noise is artificially added. In this paper, we investigate the effects that the Lombard reflex has on the performance of audio-visual speech enhancement systems based on deep learning. The results show that a gap in the performance of as much as approximately 5 dB between the systems trained on neutral speech and the ones trained on Lombard speech exists. This indicates the benefit of taking into account the mismatch between neutral and Lombard speech in the design of audio-visual speech enhancement systems.

AB - Humans tend to change their way of speaking when they are immersed in a noisy environment, a reflex known as Lombard effect. Current speech enhancement systems based on deep learning do not usually take into account this change in the speaking style, because they are trained with neutral (non-Lombard) speech utterances recorded under quiet conditions to which noise is artificially added. In this paper, we investigate the effects that the Lombard reflex has on the performance of audio-visual speech enhancement systems based on deep learning. The results show that a gap in the performance of as much as approximately 5 dB between the systems trained on neutral speech and the ones trained on Lombard speech exists. This indicates the benefit of taking into account the mismatch between neutral and Lombard speech in the design of audio-visual speech enhancement systems.

KW - Audio-visual speech enhancement

KW - Lombard effect

KW - deep learning

UR - http://www.scopus.com/inward/record.url?scp=85068971184&partnerID=8YFLogxK

U2 - 10.1109/ICASSP.2019.8682713

DO - 10.1109/ICASSP.2019.8682713

M3 - Article in proceeding

SN - 978-1-4799-8132-8

SP - 6615

EP - 6619

BT - ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

PB - IEEE

ER -

Michelsanti D, Tan Z-H, Sigurdsson S, Jensen J. Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2019. p. 6615-6619. 8682713. (I E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings). https://doi.org/10.1109/ICASSP.2019.8682713