An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits

Kai Li, Fenghua Xie, Hang Chen, Kexin Yuan, Xiaolin Hu
Tsinghua University, Beijing, China
Arxiv 2022
arXiv Code

The following are the key points of our article:
(1) A novel neural network called Cortico-Thalamo-Cortical Network (CTCNet) is proposed for audio-visual speech separation.
(2) CTCNet learns hierarchical auditory and visual representations in separate subnetworks.
(3) The model fuses auditory and visual information in a thalamic subnetwork through top-down connections.
(4) CTCNet outperforms existing AVSS methods with fewer parameters.
(5) Mimicking the anatomical connectome of the mammalian brain has great potential for advancing deep neural networks.

Abstract

Audio-visual approaches involving visual inputs have laid the foundation for recent progress in speech separation. However, the optimization of the concurrent usage of auditory and visual inputs is still an active research area. Inspired by the cortico-thalamocortical circuit, in which the sensory processing mechanisms of different modalities modulate one another via the non-lemniscal sensory thalamus, we propose a novel cortico-thalamo-cortical neural network (CTCNet) for audio-visual speech separation (AVSS). First, the CTCNet learns hierarchical auditory and visual representations in a bottom-up manner in separate auditory and visual subnetworks, mimicking the functions of the auditory and visual cortical areas. Then, inspired by the large number of connections between cortical regions and the thalamus, the model fuses the auditory and visual information in a thalamic subnetwork through top-down connections. Finally, the model transmits this fused information back to the auditory and visual subnetworks, and the above process is repeated several times. The results of experiments on three speech separation benchmark datasets show that CTCNet remarkably outperforms existing AVSS methods with considerablely fewer parameters. These results suggest that mimicking the anatomical connectome of the mammalian brain has great potential for advancing the development of deep neural networks.

Model Structures and Results

Video Presentation

BibTeX

@article{li2022audio,
        title={An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits},
        author={Li, Kai and Xie, Fenghua and Chen, Hang and Yuan, Kexin and Hu, Xiaolin},
        journal={arXiv preprint arXiv:2212.10744},
        year={2022}
      }