A Vision-Assisted Hearing Aid System Based on Deep Learning

Daniel Michelsanti; Zheng Hua Tan; Sergi Rotger-Griful; Jesper Jensen

doi:10.1109/ICASSPW59220.2023.10193370

A Vision-Assisted Hearing Aid System Based on Deep Learning

Daniel Michelsanti^*, Zheng Hua Tan, Sergi Rotger-Griful, Jesper Jensen

^*Corresponding author for this work

Research output: Contribution to book/anthology/report/conference proceeding › Article in proceeding › Research › peer-review

Abstract

Audio-visual speech enhancement (SE) is the task of reducing the acoustic background noise in a degraded speech signal using both acoustic and visual information. In this work, we study how to incorporate visual information to enhance a speech signal using acoustic beamformers in hearing aids (HAs). Specifically, we first trained a deep learning model to estimate a time-frequency mask from audio-visual data. Then, we apply this mask to estimate the inter-microphone power spectral densities (PSDs) of the clean and the noise signal. Finally, we used the estimated PSDs to build acoustic beamformers. Assuming that a HA user wears an add-on device comprising a camera pointing at the target speaker, we show that our method can be beneficial for HA systems especially at low signal to noise ratios (SNRs).

Original language	English
Title of host publication	ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings
Publisher	IEEE (Institute of Electrical and Electronics Engineers)
Publication date	2023
Article number	10193370
ISBN (Electronic)	9798350302615
DOIs	https://doi.org/10.1109/ICASSPW59220.2023.10193370
Publication status	Published - 2023
Event	2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, ICASSPW 2023 - Rhodes Island, Greece Duration: 4 Jun 2023 → 10 Jun 2023

Conference

Conference	2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, ICASSPW 2023
Country/Territory	Greece
City	Rhodes Island
Period	04/06/2023 → 10/06/2023
Sponsor	IEEE, IEEE Signal Processing Society

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

Keywords

Audio-visual
beamforming
deep learning
hearing aids

Access to Document

10.1109/ICASSPW59220.2023.10193370

AUB Link

Search for the material in Aalborg University Library's search engine

Cite this

Michelsanti, D., Tan, Z. H., Rotger-Griful, S., & Jensen, J. (2023). A Vision-Assisted Hearing Aid System Based on Deep Learning. In ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings Article 10193370 IEEE (Institute of Electrical and Electronics Engineers). https://doi.org/10.1109/ICASSPW59220.2023.10193370

@inproceedings{68d87122d03c4c89ae307419e913d82a,

title = "A Vision-Assisted Hearing Aid System Based on Deep Learning",

abstract = "Audio-visual speech enhancement (SE) is the task of reducing the acoustic background noise in a degraded speech signal using both acoustic and visual information. In this work, we study how to incorporate visual information to enhance a speech signal using acoustic beamformers in hearing aids (HAs). Specifically, we first trained a deep learning model to estimate a time-frequency mask from audio-visual data. Then, we apply this mask to estimate the inter-microphone power spectral densities (PSDs) of the clean and the noise signal. Finally, we used the estimated PSDs to build acoustic beamformers. Assuming that a HA user wears an add-on device comprising a camera pointing at the target speaker, we show that our method can be beneficial for HA systems especially at low signal to noise ratios (SNRs).",

keywords = "Audio-visual, beamforming, deep learning, hearing aids",

author = "Daniel Michelsanti and Tan, {Zheng Hua} and Sergi Rotger-Griful and Jesper Jensen",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, ICASSPW 2023 ; Conference date: 04-06-2023 Through 10-06-2023",

year = "2023",

doi = "10.1109/ICASSPW59220.2023.10193370",

language = "English",

booktitle = "ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings",

publisher = "IEEE (Institute of Electrical and Electronics Engineers)",

address = "United States",

}

Michelsanti, D, Tan, ZH, Rotger-Griful, S & Jensen, J 2023, A Vision-Assisted Hearing Aid System Based on Deep Learning. in ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings., 10193370, IEEE (Institute of Electrical and Electronics Engineers), 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, ICASSPW 2023, Rhodes Island, Greece, 04/06/2023. https://doi.org/10.1109/ICASSPW59220.2023.10193370

A Vision-Assisted Hearing Aid System Based on Deep Learning. / Michelsanti, Daniel; Tan, Zheng Hua; Rotger-Griful, Sergi et al.
ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings. IEEE (Institute of Electrical and Electronics Engineers), 2023. 10193370.

Research output: Contribution to book/anthology/report/conference proceeding › Article in proceeding › Research › peer-review

TY - GEN

T1 - A Vision-Assisted Hearing Aid System Based on Deep Learning

AU - Michelsanti, Daniel

AU - Tan, Zheng Hua

AU - Rotger-Griful, Sergi

AU - Jensen, Jesper

PY - 2023

Y1 - 2023

N2 - Audio-visual speech enhancement (SE) is the task of reducing the acoustic background noise in a degraded speech signal using both acoustic and visual information. In this work, we study how to incorporate visual information to enhance a speech signal using acoustic beamformers in hearing aids (HAs). Specifically, we first trained a deep learning model to estimate a time-frequency mask from audio-visual data. Then, we apply this mask to estimate the inter-microphone power spectral densities (PSDs) of the clean and the noise signal. Finally, we used the estimated PSDs to build acoustic beamformers. Assuming that a HA user wears an add-on device comprising a camera pointing at the target speaker, we show that our method can be beneficial for HA systems especially at low signal to noise ratios (SNRs).

AB - Audio-visual speech enhancement (SE) is the task of reducing the acoustic background noise in a degraded speech signal using both acoustic and visual information. In this work, we study how to incorporate visual information to enhance a speech signal using acoustic beamformers in hearing aids (HAs). Specifically, we first trained a deep learning model to estimate a time-frequency mask from audio-visual data. Then, we apply this mask to estimate the inter-microphone power spectral densities (PSDs) of the clean and the noise signal. Finally, we used the estimated PSDs to build acoustic beamformers. Assuming that a HA user wears an add-on device comprising a camera pointing at the target speaker, we show that our method can be beneficial for HA systems especially at low signal to noise ratios (SNRs).

KW - Audio-visual

KW - beamforming

KW - deep learning

KW - hearing aids

UR - http://www.scopus.com/inward/record.url?scp=85168247418&partnerID=8YFLogxK

U2 - 10.1109/ICASSPW59220.2023.10193370

DO - 10.1109/ICASSPW59220.2023.10193370

M3 - Article in proceeding

AN - SCOPUS:85168247418

BT - ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings

PB - IEEE (Institute of Electrical and Electronics Engineers)

T2 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, ICASSPW 2023

Y2 - 4 June 2023 through 10 June 2023

ER -

A Vision-Assisted Hearing Aid System Based on Deep Learning

Abstract

Conference

Bibliographical note

Keywords

Access to Document

AUB Link

Other files and links

Fingerprint

Cite this