An Audio-Visual Speech Enhancement System Based on 3D Image Features: An Application in Hearing Aids

Yu Ching Chung; Ji Yan Han; Bo Sin Wang; Wei Zhong Zheng; Kung Yao Shen; Ying Hui Lai

doi:10.1109/APSIPAASC58517.2023.10317139

An Audio-Visual Speech Enhancement System Based on 3D Image Features: An Application in Hearing Aids

Yu Ching Chung, Ji Yan Han, Bo Sin Wang, Wei Zhong Zheng, Kung Yao Shen, Ying Hui Lai^*

^*Corresponding author for this work

Department of Biomedical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Previous research has shown that auditory and visual inputs are not asynchronous in the human brain, and that visual cues can enhance attention in the hearing process. Therefore, this study proposes audio-visual speech enhancement (SE) with 3D image features (AV-3D-SE) that imitates the auditory process of humans to elevate listening quality. More specifically, AV-3D-SE uses the FlowNet3D model to predict temporal facial motion from the recorded 3D image combining with features for SE applications. The evaluation results showed that the average scores of perceptual evaluation of speech quality and short-time objective intelligibility in 3 dB signal-to-noise ratio increased to 3.229 and 0.914, respectively, while the average hearing aid speech quality index significantly outperformed baseline SE systems (audio-only and audio-visual-2D) in seven typical types of hearing loss with high hearing aid speech perception index. In conclusion, the proposed AV-3D-SE enhances the effectiveness of the SE system and can increase the listening satisfaction of hearing aid users.

Original language	English
Title of host publication	2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1131-1137
Number of pages	7
ISBN (Electronic)	9798350300673
DOIs	https://doi.org/10.1109/APSIPAASC58517.2023.10317139
State	Published - 2023
Event	2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023 - Taipei, Taiwan Duration: 31 Oct 2023 → 3 Nov 2023

Publication series

Name	2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023

Conference

Conference	2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023
Country/Territory	Taiwan
City	Taipei
Period	31/10/23 → 3/11/23

Keywords

deep learning
hearing aid and speech enhancement
point cloud
scene flow

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1109/APSIPAASC58517.2023.10317139

Cite this

Chung, Y. C., Han, J. Y., Wang, B. S., Zheng, W. Z., Shen, K. Y., & Lai, Y. H. (2023). An Audio-Visual Speech Enhancement System Based on 3D Image Features: An Application in Hearing Aids. In 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023 (pp. 1131-1137). (2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/APSIPAASC58517.2023.10317139

Chung, Yu Ching ; Han, Ji Yan ; Wang, Bo Sin et al. / An Audio-Visual Speech Enhancement System Based on 3D Image Features : An Application in Hearing Aids. 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023. Institute of Electrical and Electronics Engineers Inc., 2023. pp. 1131-1137 (2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023).

@inproceedings{33c0944769a24e4b8e9ba351ea1d90e3,

title = "An Audio-Visual Speech Enhancement System Based on 3D Image Features: An Application in Hearing Aids",

abstract = "Previous research has shown that auditory and visual inputs are not asynchronous in the human brain, and that visual cues can enhance attention in the hearing process. Therefore, this study proposes audio-visual speech enhancement (SE) with 3D image features (AV-3D-SE) that imitates the auditory process of humans to elevate listening quality. More specifically, AV-3D-SE uses the FlowNet3D model to predict temporal facial motion from the recorded 3D image combining with features for SE applications. The evaluation results showed that the average scores of perceptual evaluation of speech quality and short-time objective intelligibility in 3 dB signal-to-noise ratio increased to 3.229 and 0.914, respectively, while the average hearing aid speech quality index significantly outperformed baseline SE systems (audio-only and audio-visual-2D) in seven typical types of hearing loss with high hearing aid speech perception index. In conclusion, the proposed AV-3D-SE enhances the effectiveness of the SE system and can increase the listening satisfaction of hearing aid users.",

keywords = "deep learning, hearing aid and speech enhancement, point cloud, scene flow",

author = "Chung, {Yu Ching} and Han, {Ji Yan} and Wang, {Bo Sin} and Zheng, {Wei Zhong} and Shen, {Kung Yao} and Lai, {Ying Hui}",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023 ; Conference date: 31-10-2023 Through 03-11-2023",

year = "2023",

doi = "10.1109/APSIPAASC58517.2023.10317139",

language = "English",

series = "2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1131--1137",

booktitle = "2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023",

address = "美國",

}

Chung, YC, Han, JY, Wang, BS, Zheng, WZ, Shen, KY & Lai, YH 2023, An Audio-Visual Speech Enhancement System Based on 3D Image Features: An Application in Hearing Aids. in 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023. 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023, Institute of Electrical and Electronics Engineers Inc., pp. 1131-1137, 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023, Taipei, Taiwan, 31/10/23. https://doi.org/10.1109/APSIPAASC58517.2023.10317139

An Audio-Visual Speech Enhancement System Based on 3D Image Features: An Application in Hearing Aids. / Chung, Yu Ching; Han, Ji Yan; Wang, Bo Sin et al.
2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023. Institute of Electrical and Electronics Engineers Inc., 2023. p. 1131-1137 (2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - An Audio-Visual Speech Enhancement System Based on 3D Image Features

T2 - 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023

AU - Chung, Yu Ching

AU - Han, Ji Yan

AU - Wang, Bo Sin

AU - Zheng, Wei Zhong

AU - Shen, Kung Yao

AU - Lai, Ying Hui

PY - 2023

Y1 - 2023

N2 - Previous research has shown that auditory and visual inputs are not asynchronous in the human brain, and that visual cues can enhance attention in the hearing process. Therefore, this study proposes audio-visual speech enhancement (SE) with 3D image features (AV-3D-SE) that imitates the auditory process of humans to elevate listening quality. More specifically, AV-3D-SE uses the FlowNet3D model to predict temporal facial motion from the recorded 3D image combining with features for SE applications. The evaluation results showed that the average scores of perceptual evaluation of speech quality and short-time objective intelligibility in 3 dB signal-to-noise ratio increased to 3.229 and 0.914, respectively, while the average hearing aid speech quality index significantly outperformed baseline SE systems (audio-only and audio-visual-2D) in seven typical types of hearing loss with high hearing aid speech perception index. In conclusion, the proposed AV-3D-SE enhances the effectiveness of the SE system and can increase the listening satisfaction of hearing aid users.

AB - Previous research has shown that auditory and visual inputs are not asynchronous in the human brain, and that visual cues can enhance attention in the hearing process. Therefore, this study proposes audio-visual speech enhancement (SE) with 3D image features (AV-3D-SE) that imitates the auditory process of humans to elevate listening quality. More specifically, AV-3D-SE uses the FlowNet3D model to predict temporal facial motion from the recorded 3D image combining with features for SE applications. The evaluation results showed that the average scores of perceptual evaluation of speech quality and short-time objective intelligibility in 3 dB signal-to-noise ratio increased to 3.229 and 0.914, respectively, while the average hearing aid speech quality index significantly outperformed baseline SE systems (audio-only and audio-visual-2D) in seven typical types of hearing loss with high hearing aid speech perception index. In conclusion, the proposed AV-3D-SE enhances the effectiveness of the SE system and can increase the listening satisfaction of hearing aid users.

KW - deep learning

KW - hearing aid and speech enhancement

KW - point cloud

KW - scene flow

UR - http://www.scopus.com/inward/record.url?scp=85180005823&partnerID=8YFLogxK

U2 - 10.1109/APSIPAASC58517.2023.10317139

DO - 10.1109/APSIPAASC58517.2023.10317139

M3 - Conference contribution

AN - SCOPUS:85180005823

T3 - 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023

SP - 1131

EP - 1137

BT - 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 31 October 2023 through 3 November 2023

ER -

Chung YC, Han JY, Wang BS, Zheng WZ, Shen KY, Lai YH. An Audio-Visual Speech Enhancement System Based on 3D Image Features: An Application in Hearing Aids. In 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023. Institute of Electrical and Electronics Engineers Inc. 2023. p. 1131-1137. (2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023). doi: 10.1109/APSIPAASC58517.2023.10317139

An Audio-Visual Speech Enhancement System Based on 3D Image Features: An Application in Hearing Aids

Abstract

Publication series

Conference

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this