Changchun Bao

In multimodal emotion recognition, the diversity and temporal unalignment of speech and text modalities pose significant challenges for effective fusion. To address this issue, Multi-level Interaction for Emotion Recognition from Unaligned Speech and Text (MIUST) is proposed. Inspired by the hierarchical and multi-level integration process of human emotion cognition, the MIUST framework is designed to be consisted of a unimodal emotion recognition module, a multi-level cross-modal fusion module incorporating both coarse-grained feature learning and fine-grained modal fusion (FGMF), and an emotion classification module that synthesizes information from all stages. This multi-level branched fusion architecture more closely mirrors the human emotion understanding process. In addition, the FGMF can achieve cross-granularity fusion of speech and text. It learns emotional correlations between individual speech frames and textual words, thereby eliminating the need for explicit temporal alignment between modalities. Experimental results on the IEMOCAP and MELD datasets demonstrate the effectiveness of MIUST, achieving 77.78% weighted accuracy (WA), 78.79% unweighted accuracy (UA), and 77.82% weighted F1-score (W-F1) on IEMOCAP, outperforming existing state-of-the-art methods. These results validate that MIUST effectively improves multimodal emotion recognition performance by leveraging multi-level and cross-modal feature interaction. Our code is available at https://github.com/HANLM15/MIUST.

In this paper, we propose a speech enhancement method us ing dual-path Multi-Channel Linear Prediction (MCLP) filters and multi-norm beamforming. Specifically, the MCLP part in the proposed method is designed with dual-path filters in both time and frequency dimensions. For the beamforming part, we minimize the power of the microphone array output as well as the l1 norm of the denoised signals while preserving source sig nals from the target directions. An efficient method to select the prediction orders in the dual-path filters is also proposed, which is robust for signals with different reverberation time (T60) val ues and can be applied to other MCLP-based methods. Eval uations demonstrate that our proposed method outperforms the baseline methods for speech enhancement, particularly in high reverberation scenarios.

This study proposes a three-dimensional room transfer function (RTF) parameterization method based on multiple concentric planar circular arrays, which exhibits robustness to variations in the positions of both the receiver and source. According to the harmonic solution to the wave equation, the RTFs between two spherical regions (sound source and receiver) in a room can be expressed as a weighted sum of spherical harmonics, whose weight coefficients serve as the RTF parameters, which can be estimated by placing multiple concentric planar circular arrays composed of monopole-source pairs (MSPs) and multiple concentric planar circular arrays composed of omnidirectional-microphone pairs (OMPs) in respective source and receiver regions. We use MSP arrays to generate required outgoing soundfields originating from a source region. We derive a method to use OMP arrays to estimate RTF parameters that are concealed within the captured soundfield, which can be employed to reconstruct the RTF from any point in the source region to any point in the receiver region. The accuracy of the RTF parameterization method is validated through simulation testing.

As a practical way to address the cocktail party problem, the target speaker extraction (TSE) has received increasing attention recently. Typically, the TSE is explored in two scenarios. The first scenario is a specific one, where the target speaker is present and the signal received by the microphone contains at least two speakers. The second scenario is a universal one, where the presence or absence of the target speaker is uncertain and the received signal may contain one or multiple speakers. For these two scenarios, numerous TSE studies rely on the speaker embedding derived from an enrollment of the target speaker to guide the extraction. However, solely utilizing this embedding may not fully leverage the contextual information within the enrollment. To address this limitation, a novel approach that directly exploits the contextual information in the time-frequency (T-F) domain was recently proposed. This paper improves this approach by integrating our previously proposed coarse-to-fine framework, which consists of a coarse extraction and a refining extraction. For the specific scenario, an interaction block with an attention mechanism is employed to facilitate direct interaction between the T-F representations of the enrollment and the received signal. This direct interaction leads to the consistent T-F representation of the enrollment that serves as guidance for the coarse extraction. Afterwards, the T-F representation of the coarsely extracted signal, which may contain some distortions, is utilized to guide the refining extraction. The residual T-F representation obtained during the refining extraction mitigates distortions of coarse extraction and increases the precision of speaker extraction. Besides, this paper explores an undisturbed version of the universal scenario where the noise and reverberation are not considered. An additional processor is integrated and a two-level decision-making scheme is devised to generalize the coarse-tofine TSE method for this undisturbed universal scenario. The proposed method achieves the state-of-the-art performance on different benchmark datasets for the specific scenario and is proven effective for the undisturbed universal scenario.

The combination of neural networks and beamforming has proven very effective in multi-channel speech separation, but its performance faces a challenge in complex environments. In this paper, an iteratively refined multi-channel speech separation method is proposed to meet this challenge. The proposed method is composed of initial separation and iterative separation. In the initial separation, a time–frequency domain dual-path recurrent neural network (TFDPRNN), minimum variance distortionless response (MVDR) beamformer, and post-separation are cascaded to obtain the first additional input in the iterative separation process. In iterative separation, the MVDR beamformer and post-separation are iteratively used, where the output of the MVDR beamformer is used as an additional input to the post-separation network and the final output comes from the post-separation module. This iteration of the beamformer and post-separation is fully employed for promoting their optimization, which ultimately improves the overall performance. Experiments on the spatialized version of the WSJ0-2mix corpus showed that our proposed method achieved a signal-to-distortion ratio (SDR) improvement of 24.17 dB, which was significantly better than the current popular methods. In addition, the method also achieved an SDR of 20.2 dB on joint separation and dereverberation tasks. These results indicate our method’s effectiveness and significance in the multi-channel speech separation field.

Automatic music transcription (AMT) is to transcribe music audio into note symbol representations. Concurrent notes overlapping in the frequency and time domains still hinder the performance of polyphonic piano transcription in current studies. In this work, we develop an attention-based method for piano transcription, where we propose a harmonic-aware attention to capture the musical frequency structure, and a local time attention to model temporal dependencies. The harmonic-aware frequency attention not only emphasizes the relationship between the obvious harmonics, but also extracts the correlation in the residual non-harmonic component. The time attention mechanism is improved using the learnable attention range masks to model frame-wise short-term dependencies on different subtasks. Experiments on the MAESTRO dataset demonstrate that the proposed system achieves state-of-the-art transcription performance on both frame-wise and note-wise F1 metrics. Considering the influence of the piano pedals' dynamic behavior on note duration, a note duration modification method is also proposed. With a more accurate annotation of the offset on MAESTRO, the transcription performance is further improved.

As a research focus within the field of array signal processing, multi-source direction-of-arrival (DOA) estimation in enclosed environments has been paid much attention. Contaminated by reverberation, noise, and inter-source interference, DOA estimation become challenging. Hence it is essential to identify time-frequency (TF) points dominated by only one source to alleviate these issues. This paper proposes a TF point selection method for DOA estimation based on the first-order relative harmonic coefficient (RHC). This is first analyzed on the “point” level from two perspective, and we design an adaptive single-source dominant zone (SSDZ) detection method. Subsequently, the relationship between first- and zero-order RHC magnitudes of different types of TF points is explored, and we develop a simple but useful rule to further select TF points in the detected SSDZs. Finally, we adopt two-dimensional (2-D) kernel density estimation (KDE) and peak search to estimate the DOAs of sources after calculating the angles of the detected TF points. The effectiveness and robustness of the proposed method are verified and compared with the reference methods through experiments with both the simulated and real-world recordings.

The combination of neural network and beamforming has been proved to be very effective in multi-channel speech separation. But its performance faces a challenge in complex environment. In this paper, an iteratively refined multi-channel speech separation method is proposed for the challenge, where the proposed is composed of initial separation and iterative separation. In initial separation, the time-frequency domain dual-path recurrent neural network neural network (TFDPRNN), minimum variance distortionless response (MVDR) beamformer and post-separation (also TFDPRNN) are cascaded for obtaining the first additional input in iterative separation. In iterative separation, the MVDR beamformer and post-separation are iteratively used, where the output of the MVDR beamformer is used as an additional input of the post-separation network and the final output comes from post-separation module. This iteration of the beamformer and post-separation is fully employed for promoting their individual optimization, which ultimately improves the overall performance of speech separation in multi-speaker scenarios. Experiments on the spatialized version of the WSJ0-2mix corpus show that our proposed method is significantly better than the current popular methods. In addition, the method also has a good effect on the dereverberation task.

Dynamic parameterization of acoustic environments has drawn widespread attention in the field of audio processing. Precise representation of local room acoustic characteristics is crucial when designing audio filters for various audio rendering applications. Key parameters in this context include reverberation time (RT60) and geometric room volume. In recent years, neural networks have been extensively applied in the task of blind room parameter estimation. However, there remains a question of whether pure attention mechanisms can achieve superior performance in this task. To address this issue, this study employs blind room parameter estimation based on monaural noisy speech signals. Various model architectures are investigated, including a proposed attention-based model. This model is a convolution-free Audio Spectrogram Transformer, utilizing patch splitting, attention mechanisms, and cross-modality transfer learning from a pretrained Vision Transformer. Experimental results suggest that the proposed attention mechanism-based model, relying purely on attention mechanisms without using convolution, exhibits significantly improved performance across various room parameter estimation tasks, especially with the help of dedicated pretraining and data augmentation schemes. Additionally, the model demonstrates more advantageous adaptability and robustness when handling variable-length audio inputs compared to existing methods.

In target speaker extraction, many studies rely on the speaker embedding which is obtained from an enrollment of the target speaker and employed as the guidance. However, solely using speaker embedding may not fully utilize the contextual information contained in the enrollment. In this paper, we directly exploit this contextual information in the time-frequency (T-F) domain. Specifically, the T-F representations of the enrollment and the mixed signal are interacted to compute the weighting matrices through an attention mechanism. These weighting matrices reflect the similarity among different frames of the T-F representations and are further employed to obtain the consistent T-F representations of the enrollment. These consistent representations are served as the guidance, allowing for better exploitation of the contextual information. Furthermore, the proposed method achieves the state-of-the-art performance on the benchmark dataset and shows its effectiveness in the complex scenarios.