Decoding neural representations of rhythmic sounds from magnetoencephalography

Pei Chun Chang, Jia Ren Chang, Po Yu Chen, Li Kai Cheng, Jen-Chuen Hsieh, Hsin Yen Yu, Li-Fen Chen, Yong-Sheng Chen

Research output: Contribution to journalConference articlepeer-review

Abstract

Neuroscience studies have revealed neural processes involving rhythm perception, suggesting that brain encodes rhythmic sounds and embeds information in neural activity. In this work, we investigate how to extract rhythmic information embedded in the brain responses and to decode the original audio waveforms from the extracted information. A spatiotemporal convolutional neural network is adopted to extract compact rhythm-related representations from the noninvasively measured magnetoencephalographic (MEG) signals evoked by listening to rhythmic sounds. These learned MEG representations are then used to condition an audio generator network for the synthesis of the original rhythmic sounds. In the experiments, we evaluated the proposed method by using the MEG signals recorded from eight participants and demonstrated that the generated rhythms are highly related to those evoking the MEG signals. Interestingly, we found that the auditory-related MEG channels reveal high importance in encoding rhythmic representations, the distribution of these representations relate to the timing of beats, and the behavior performance is consistent with the performance of neural decoding. These results suggest that the proposed method can synthesize rhythms by decoding neural representations from MEG.

Original languageEnglish
Pages (from-to)1280-1284
Number of pages5
JournalICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Volume2021-June
DOIs
StatePublished - Jun 2021
Event2021 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2021 - Virtual, Toronto, Canada
Duration: 6 Jun 202111 Jun 2021

Keywords

  • CNN
  • GAN
  • MEG
  • Rhythm generation

Fingerprint

Dive into the research topics of 'Decoding neural representations of rhythmic sounds from magnetoencephalography'. Together they form a unique fingerprint.

Cite this