Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Rongfeng Li has a total of 27 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Pipa-Multi-TechSpot: Automatic Annotation of Pipa Playing Techniques Using Multimodal Information and Expert Rules

    by Rongfeng Li, Zilong Ke, Bo Qiu

    Pipa playing techniques involve string contact, hand movement, and acoustic characteristics that cannot be fully described by note information or a single modality. We present a framework for automatic annotation of pipa playing techniques that combines audiovisual information with expert rules. Synchronized recordings and basic fingering annotations establish temporal and spatial links among audiovisual segments, string and fret positions, and note events. Acoustic and visual modules are then organized according to the movement characteristics of individual techniques. The acoustic branch uses envelope spectra to characterize plucking rhythms and augments an Audio Spectrogram Transformer with patch mixing and focal loss. The visual branch identifies techniques from the displacement, direction, and oscillation of hand keypoints. Expert rules describing technique combinations and temporal relationships subsequently check the initial labels in context; correction benefits are assessed by counting both corrected errors and newly introduced errors. Experiments on a pipa performance dataset yield overall accuracies of 83.2% for right-hand techniques and 64.6% for left-hand techniques over all annotated segments. Because the left-hand categories depend on movement that the video must resolve, left-hand accuracy is additionally reported on the evaluable subset defined by the input conditions of the visual pathway: 0.837 of left-hand annotations meet these conditions, and accuracy on this subset is 0.755. Expert rules produce positive net correction gains for both hands. These results support organizing audiovisual evidence around performance actions and incorporating performance knowledge into annotation, providing a methodological basis for fine-grained pipa datasets and subsequent performance analysis.

  • Open Access

    Expressive Dizi Synthesis: Unlimited Synthetic Datasets, Real–Synthetic Integration and Technique Labeling

    by Rongfeng Li, Junchen Liu, Zijin Li, Ya Li, Linfeng Fan, Pei Huang

    Digital modeling of traditional Chinese musical instruments significantly lags behind that of their Western counterparts, limiting advances in cultural preservation and related research. This paper addresses score-to-audio generation for the Chinese bamboo flute (dizi), aiming to synthesize expressive audio with human-like performance nuances directly from Musical Instrument Digital Interface (MIDI) scores. Two core challenges remain in existing score-to-audio synthesis methods: first, the scarcity of large-scale paired MIDI-audio training data for traditional Chinese instruments; second, the inability of standard MIDI to encode instrument-specific expressive techniques, such as vibrato, pitch bends, and ornamentations. To address these challenges, we propose a scalable workflow that generates large-scale synthetic MIDI-audio pairs through rule-based score randomization, automated digital audio workstation (DAW) rendering with commercial sample libraries, and standardized feature extraction. Performance technique information is directly encoded into the MIDI stream using out-of-range MIDI note numbers, achieving more effective conditioning than external control methods. Our system is built upon the Musical Instrument Digital Interface–Differentiable Digital Signal Processing (MIDI-DDSP) framework. Large-scale synthetic data is used for pre-training to establish timbral consistency, while real recordings from the University of Rochester Multi-Modal Music Performance (URMP) flute corpus are employed for fine adjustment to achieve expressive dynamic variations. Synthetic data alone yields stable but less expressive outputs, whereas real data alone risks overfitting. The combined strategy achieves a realistic timbre with controllable dynamics and performance techniques.

  • Open Access

    Expressive Dizi Synthesis: Unlimited Synthetic Datasets, Real–Synthetic Integration and Technique Labeling

    by Rongfeng Li, Junchen Liu, Zijin Li, Ya Li, Linfeng Fan, Pei Huang

    The digitized models of Chinese traditional musical instruments have fallen behind those of Western musical instruments, therefore they have restricted the preservation work and the research acquiring. This article deals with score-to-audio generation for the Chinese bamboo flute (dizi), it produces expressive audio from MIDI scores which has human performance detail. The current existing methods have two obstacles that they face. First, the large-scale paired training data (MIDI-audio) for traditional Chinese musical instruments is not much. Second, the already existing systems are not able to control the techniques that are specific to instruments—vibrato, pitch bends, ornaments—which the standard MIDI has no ability to encode. We put forward a expandable working flow that produces infinite artificial MIDI-audio matching groups through rule-based music score random arrangement, automatic DAW drawing output by using business sample storage banks, and systemized character pick-up. The information of technique is directly coded in the MIDI stream through the use of note numbers that lie outside the normal range, which thus proves to be more effective than the method of external conditioning. This system is constructed on the basis of MIDI-DDSP. Pre-training makes use of large-scale man-made data to obtain timbre consistency; The fine adjustment carries out the use of actual recording materials (URMP flute corpus) to realize expressive dynamic changes. Only the synthetic data can generate output that is stable but has no life. The real data which exist alone can bring about the problem of overfitting. This combined result can give realistic tone, which has controllable dynamic changes and playing skills.

  • Open Access

    Scalable Synthetic Data Pipeline and End-to-End Sound Synthesis: A Case Study on the Chinese Dizi

    by Junchen Liu, Rongfeng Li, Zijin Li, Ya Li, Linfeng Fan, Pei Huang

  • Open Access

    Intelligent Installation Design and Implementation of Automatic Performative Transcription for Plucked String Instrument

    by Bo Qiu, Rongfeng Li, Linfeng Fan

    Since the 20th century, synthesizers and MIDI keyboards offer composers more convenient operation and a higher creative ceiling. However, traditional Chinese plucked instruments like the Pipa have only just begun to explore the realms of electronic music and automatic notation. This study focuses on the Pipa as the subject, aiming to design and implement a device that captures performance-related data while maintaining the original conditions of the instrument, and automatically transcribes this data into musical notation.This study presents a data collection module based on Arduino hardware. This design involves detection by making certain modifications to the instrument and connecting wires to the instrument's frets which achieves real-time detection of the performer's finger movements via a hardware loop, collecting performance data for transcription. This setup allows performer to record and preserve details of their performance, including the fingering and string values for each note played. Through extensive research on sensors and underlying theories, this study has essentially achieved an intelligent device for the automatic notation of plucked instrument performances. Furthermore, with further refinements, it can also be applied to more complex,technically rich musical compositions and more instruments in the future.

  • Open Access

    Pipa Performance Generation Based on Pre-trained Temporally Guided Network

    by Jiaxuan Song, Rongfeng Li

    In this paper, we propose a method that takes the audio of a solo pipa performance as input and utilizes pre-trained models of other instruments to enhance the pipa's 3D skeleton motion performance. This approach aims to address the limited availability of training data. The key idea is that we believe that all instruments exhibit similar motion trends in their large limbs while playing. Therefore, we can utilize videos of other instruments playing to pre-train and capture the motion trends of the large limbs. Subsequently, we can fine-tune the model using videos of the pipa solo, thereby addressing the issue of limited data for the pipa solo. We utilize a combination of LSTM network, U-net architecture and self-attention mechanism to validate the effectiveness of instrumental pre-training using videos of pipa solos on the Internet.

  • Open Access

    Singing-Voice Timbre Evaluations Based on Transfer Learning

    by Rongfeng Li, Mingtong Zhang

    The development of artificial intelligence technology has made it possible to realize automatic evaluation systems for singing, and relevant research has been able to achieve accurate evaluations with respect to pitch and rhythm, but research on singing-voice timbre evaluation has remained at the level of theoretical analysis. Timbre is closely related to expression performance, breath control, emotional rendering, and other aspects of singing skills, and it has a crucial impact on the evaluation of song interpretation. The purpose of this research is to investigate the automatic evaluation method of singing-voice timbre. At the present stage, timbre research generally has problems such as a paucity of datasets, a single evaluation index, easy overfitting or a model’s failure to converge. Compared with the singing voice, the research on musical instruments is more mature, with more available data and richer evaluation dimensions. We constructed a deep network based on the CRNN model to perform timbre evaluation, and the test results showed that cross-media learning of timbre evaluation is feasible, which also indicates that humans have a consistent timbre perception with respect to musical instruments and vocals.