Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems

Daniel Michelsanti; Zheng-Hua Tan; Sigurdur Sigurdsson; Jesper Jensen

doi:10.1109/ICASSP.2019.8682713

Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems

Daniel Michelsanti, Zheng-Hua Tan, Sigurdur Sigurdsson, Jesper Jensen

Research output: Contribution to book/anthology/report/conference proceeding › Article in proceeding › Research › peer-review

6 Citations (Scopus)

Abstract

Humans tend to change their way of speaking when they are immersed in a noisy environment, a reflex known as Lombard effect. Current speech enhancement systems based on deep learning do not usually take into account this change in the speaking style, because they are trained with neutral (non-Lombard) speech utterances recorded under quiet conditions to which noise is artificially added. In this paper, we investigate the effects that the Lombard reflex has on the performance of audio-visual speech enhancement systems based on deep learning. The results show that a gap in the performance of as much as approximately 5 dB between the systems trained on neutral speech and the ones trained on Lombard speech exists. This indicates the benefit of taking into account the mismatch between neutral and Lombard speech in the design of audio-visual speech enhancement systems.

Original language	English
Title of host publication	ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Number of pages	5
Publisher	IEEE (Institute of Electrical and Electronics Engineers)
Publication date	17 Apr 2019
Pages	6615-6619
Article number	8682713
ISBN (Print)	978-1-4799-8132-8
ISBN (Electronic)	978-1-4799-8131-1
DOIs	https://doi.org/10.1109/ICASSP.2019.8682713
Publication status	Published - 17 Apr 2019
Event	2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) - Brighton, United Kingdom Duration: 12 May 2019 → 17 May 2019 https://2019.ieeeicassp.org/

Conference

Conference	2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Country/Territory	United Kingdom
City	Brighton
Period	12/05/2019 → 17/05/2019
Internet address	https://2019.ieeeicassp.org/

Series	I E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings
ISSN	1520-6149

Keywords

Audio-visual speech enhancement
Lombard effect
deep learning

Access to Document

10.1109/ICASSP.2019.8682713

https://arxiv.org/pdf/1811.06250.pdf

AUB Link

Search for the material in Aalborg University Library's search engine

Cite this

Michelsanti, D., Tan, Z.-H., Sigurdsson, S., & Jensen, J. (2019). Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6615-6619). Article 8682713 IEEE (Institute of Electrical and Electronics Engineers). https://doi.org/10.1109/ICASSP.2019.8682713

Michelsanti, Daniel ; Tan, Zheng-Hua ; Sigurdsson, Sigurdur et al. / Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems. ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE (Institute of Electrical and Electronics Engineers), 2019. pp. 6615-6619 (I E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings).

@inproceedings{17da3cfcd9684373acc52ccbe20201fc,

title = "Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems",

abstract = "Humans tend to change their way of speaking when they are immersed in a noisy environment, a reflex known as Lombard effect. Current speech enhancement systems based on deep learning do not usually take into account this change in the speaking style, because they are trained with neutral (non-Lombard) speech utterances recorded under quiet conditions to which noise is artificially added. In this paper, we investigate the effects that the Lombard reflex has on the performance of audio-visual speech enhancement systems based on deep learning. The results show that a gap in the performance of as much as approximately 5 dB between the systems trained on neutral speech and the ones trained on Lombard speech exists. This indicates the benefit of taking into account the mismatch between neutral and Lombard speech in the design of audio-visual speech enhancement systems.",

keywords = "Audio-visual speech enhancement, Lombard effect, deep learning",

author = "Daniel Michelsanti and Zheng-Hua Tan and Sigurdur Sigurdsson and Jesper Jensen",

year = "2019",

month = apr,

day = "17",

doi = "10.1109/ICASSP.2019.8682713",

language = "English",

isbn = "978-1-4799-8132-8",

series = "I E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings",

publisher = "IEEE (Institute of Electrical and Electronics Engineers)",

pages = "6615--6619",

booktitle = "ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)",

address = "United States",

note = "2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP ; Conference date: 12-05-2019 Through 17-05-2019",

url = "https://2019.ieeeicassp.org/",

}

Michelsanti, D, Tan, Z-H, Sigurdsson, S & Jensen, J 2019, Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems. in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)., 8682713, IEEE (Institute of Electrical and Electronics Engineers), I E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings, pp. 6615-6619, 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, United Kingdom, 12/05/2019. https://doi.org/10.1109/ICASSP.2019.8682713

Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems. / Michelsanti, Daniel; Tan, Zheng-Hua; Sigurdsson, Sigurdur et al.
ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE (Institute of Electrical and Electronics Engineers), 2019. p. 6615-6619 8682713 (I E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings).

Research output: Contribution to book/anthology/report/conference proceeding › Article in proceeding › Research › peer-review

TY - GEN

T1 - Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems

AU - Michelsanti, Daniel

AU - Tan, Zheng-Hua

AU - Sigurdsson, Sigurdur

AU - Jensen, Jesper

PY - 2019/4/17

Y1 - 2019/4/17

N2 - Humans tend to change their way of speaking when they are immersed in a noisy environment, a reflex known as Lombard effect. Current speech enhancement systems based on deep learning do not usually take into account this change in the speaking style, because they are trained with neutral (non-Lombard) speech utterances recorded under quiet conditions to which noise is artificially added. In this paper, we investigate the effects that the Lombard reflex has on the performance of audio-visual speech enhancement systems based on deep learning. The results show that a gap in the performance of as much as approximately 5 dB between the systems trained on neutral speech and the ones trained on Lombard speech exists. This indicates the benefit of taking into account the mismatch between neutral and Lombard speech in the design of audio-visual speech enhancement systems.

AB - Humans tend to change their way of speaking when they are immersed in a noisy environment, a reflex known as Lombard effect. Current speech enhancement systems based on deep learning do not usually take into account this change in the speaking style, because they are trained with neutral (non-Lombard) speech utterances recorded under quiet conditions to which noise is artificially added. In this paper, we investigate the effects that the Lombard reflex has on the performance of audio-visual speech enhancement systems based on deep learning. The results show that a gap in the performance of as much as approximately 5 dB between the systems trained on neutral speech and the ones trained on Lombard speech exists. This indicates the benefit of taking into account the mismatch between neutral and Lombard speech in the design of audio-visual speech enhancement systems.

KW - Audio-visual speech enhancement

KW - Lombard effect

KW - deep learning

UR - http://www.scopus.com/inward/record.url?scp=85068971184&partnerID=8YFLogxK

U2 - 10.1109/ICASSP.2019.8682713

DO - 10.1109/ICASSP.2019.8682713

M3 - Article in proceeding

SN - 978-1-4799-8132-8

T3 - I E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings

SP - 6615

EP - 6619

BT - ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

PB - IEEE (Institute of Electrical and Electronics Engineers)

T2 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Y2 - 12 May 2019 through 17 May 2019

ER -

Michelsanti D, Tan ZH, Sigurdsson S, Jensen J. Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE (Institute of Electrical and Electronics Engineers). 2019. p. 6615-6619. 8682713. (I E E E International Conference on Acoustics, Speech and Signal Processing. Proceedings). doi: 10.1109/ICASSP.2019.8682713

Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems

Abstract

Conference

Keywords

Access to Document

AUB Link

Other files and links

Fingerprint

Cite this