SPMamba: State-space model is all you need in speech separation
━ Arxiv 2024 ━

Kai Li

Guo Chen

Xiaolin Hu

Tsinghua University 🐍

[arXiv 📝]

[code ⚙️]

[poster 🖼️]

[Echo2Mix 🗂️]

The bidirectional Mamba layers process both forward and backward sequences, allowing SPMamba to use both past and future information, which improves separation performance.

Abstract

Existing speech separation models based on LSTM and Transformer face the high complexity of long audio that is difficult to model effectively. To address this challenges, we introduce an innovative speech separation method called SPMamba. This model builds upon the robust TF-GridNet architecture, replacing its traditional BLSTM components with bidirectional Mamba modules. These modules effectively model the spatiotemporal relationships between the time and frequency dimensions, allowing SPMamba to capture long-range dependencies with linear computational complexity. Specifically, the bidirectional processing within the Mamba modules enables the model to utilize both past and future contextual information, thereby enhancing separation performance. Extensive experiments conducted on public datasets, including WSJ0-2Mix, WHAM!, and Libri2Mix, as well as the newly constructed Echo2Mix dataset, demonstrated that SPMamba significantly outperformed existing state-of-the-art models, achieving superior results while also reducing computational complexity. These findings highlighted the effectiveness of SPMamba in tackling the intricate challenges of speech separation in complex environments.

An overview of the proposed SPMamba model and BMamba layer. The BMamba layer processes both forward and backward audio sequences. "LN" denotes the layer normalization, and "CLN" denotes the cumulative layer normalization..

Comparison with State-of-the-art Models

The experimental results demonstrated that the proposed SPMamba method achieved the SOTA performance across multiple metrics. On the more complex Echo2Mix dataset, SPMamba achieved an SDRi of 16.1 dB and an SI-SNRi of 15.3 dB , setting the current state-of-the-art (SOTA) performance and significantly improving over other traditional methods.

Complexity of SPMamba and TF-GridNet

SPMamba significantly outperformed TF-GridNet regarding GPU memory usage and inference time, demonstrating the efficiency of SPMamba in long speech separation tasks.

Audio Demo

Demo One

Index	Ground Truth	SPMamba	TF-GridNet	BSRNN	TDANet	A-FRCNN	SuDORM-RF	Conv-TasNet
SPK A
SPK B

Demo Two

Index	Ground Truth	SPMamba	TF-GridNet	BSRNN	TDANet	A-FRCNN	SuDORM-RF	Conv-TasNet
SPK A
SPK B

Demo Three

Index	Ground Truth	SPMamba	TF-GridNet	BSRNN	TDANet	A-FRCNN	SuDORM-RF	Conv-TasNet
SPK A
SPK B

Demo Four

Index	Ground Truth	SPMamba	TF-GridNet	BSRNN	TDANet	A-FRCNN	SuDORM-RF	Conv-TasNet
SPK A
SPK B

Demo Five

Index	Ground Truth	SPMamba	TF-GridNet	BSRNN	TDANet	A-FRCNN	SuDORM-RF	Conv-TasNet
SPK A
SPK B

➥ BibTeX:

			
				@article{li2024spmamba,

				      title={Spmamba: State-space model is all you need in speech separation},

				      author={Li, Kai and Chen, Guo and Hu, Xiaolin},

				      year={2024},

				      journal={arXiv preprint arXiv:2404.02063}

				   }

Acknowledgements

Website template was borrowed from Colorful Image Colorization and Nerfies; the code can be found here and here. Thank you (.❛ ᴗ ❛.).

▶ cslikai.cn's clustrmaps 🌎.