SFSen Fang

Sen Fang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Sen Fang has a total of 6 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Echotune: A Modular Extractor Leveraging the Variable-Length Nature of Speech in ASR Tasks

    by Sizhou Chen, Songyang Gao, Sen Fang

    The Transformer architecture has proven to be highly effective for Automatic Speech Recognition (ASR) tasks, becoming a foundational component for a plethora of research in the domain. Historically, many approaches have leaned on fixed-length attention windows, which becomes problematic for varied speech samples in duration and complexity, leading to data over-smoothing and neglect of essential long-term connectivity. Addressing this limitation, we introduce Echo-MSA, a nimble module equipped with a variable-length attention mechanism that accommodates a range of speech sample complexities and durations. This module offers the flexibility to extract speech features across various granularities, spanning from frames and phonemes to words and discourse. The proposed design captures the variable length feature of speech and addresses the limitations of fixed-length attention. Our evaluation leverages a parallel attention architecture complemented by a dynamic gating mechanism that amalgamates traditional attention with the Echo-MSA module output. Empirical evidence from our study reveals that integrating Echo-MSA into the primary model's training regime significantly enhances the word error rate (WER) performance, all while preserving the intrinsic stability of the original model.

  • Exploring Efficient-Tuned Learning Audio Representation Method from BriVL

    by Sen Fang, Yangjian Wu, Bowen Gao, Jingwen Cai, Teik Toe Teoh


  • Open Access

    UniBriVL: Robust Universal Representation and Generation of Audio Driven Diffusion Models

    by Sen Fang, Bowen Gao, Yangjian Wu, Teik Toe Teoh

    Multimodal large models have been recognized for their advantages in various performance and downstream tasks. The development of these models is crucial towards achieving general artificial intelligence in the future. In this paper, we propose a novel universal language representation learning method called UniBriVL, which is based on Bridging-Vision-and-Language (BriVL). Universal BriVL embeds audio, image, and text into a shared space, enabling the realization of various multimodal applications. Our approach addresses major challenges in robust language (both text and audio) representation learning and effectively captures the correlation between audio and image. Additionally, we demonstrate the qualitative evaluation of the generated images from UniBriVL, which serves to highlight the potential of our approach in creating images from audio. Overall, our experimental results demonstrate the efficacy of UniBriVL in downstream tasks and its ability to choose appropriate images from audio. The proposed approach has the potential for various applications such as speech recognition, music signal processing, and captioning systems.

  • Open Access

    Exploring Efficient-Tuned Learning Audio Representation Method from BriVL

    by Sen Fang, Yangjian Wu, Bowen Gao, Jingwen Cai, Teik Toe Teoh

    Recently, researchers have gradually realized that in some cases, the self-supervised pre-training on large-scale Internet data is better than that of high-quality/manually labeled data sets, and multimodal/large models are better than single or bimodal/small models. In this paper, we propose a robust audio representation learning method WavBriVL based on Bridging-Vision-and-Language (BriVL). WavBriVL projects audio, image and text into a shared embedded space, so that multi-modal applications can be realized. We demonstrate the qualitative evaluation of the image generated from WavBriVL as a shared embedded space, with the main purposes of this paper:(1) Learning the correlation between audio and image;(2) Explore a new way of image generation, that is, use audio to generate pictures. Experimental results show that this method can effectively generate appropriate images from audio.

  • Open Access

    UniBriVL: Robust Audio Representation and Generation of Audio Driven Diffusion Models

    by Sen Fang, Bowen Gao, Yangjian Wu, TeikToe Teoh

    Sen Fang, Bowen Gao, Yangjian Wu, TeikToe Teoh. Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL). 2023.