Learning Stage-Wise GANs for Whistle Extraction in Time-Frequency Spectrograms
- 1 March 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 9302-9314
Abstract
Whistle contour extraction aims to derive animal whistles from time-frequency spectrograms as polylines. For toothed whales, whistle extraction results can serve as the basis for analyzing animal abundance, species identity, and social activities. During the last few decades, as long-term recording systems have become affordable, automated whistle extraction algorithms were proposed to process large volumes of recording data. Recently, a deep learning-based method demonstrated superior performance in extracting whistles under varying noise conditions. However, training such networks requires a large amount of labor-intensive annotation, which is not available for many species. To overcome this limitation, we present a framework of stage-wise generative adversarial networks (GANs), which compile new whistle data suitable for deep model training via three stages: generation of background noise in the spectrogram, generation of whistle contours, and generation of whistle signals. By separating the generation of different components in the samples, our framework composes visually promising whistle data and labels even when few expert annotated data are available. Regardless of the amount of human-annotated data, the proposed data augmentation framework leads to a consistent improvement in performance of the whistle extraction model, with a maximum increase of 1.69 in the whistle extraction mean F1-score. Our stage-wise GAN also surpasses one single GAN in improving whistle extraction models with augmented data. The data and code will be available at https://github.com/Paul-LiPu/CompositeGAN_WhistleAugment .Keywords
All Related Versions
Funding Information
- Michael Weise
- Office of Naval Research (N000141712867, N000142112567)
This publication has 53 references indexed in Scilit:
- Automated segmentation of linear time-frequency representations of marine-mammal soundsThe Journal of the Acoustical Society of America, 2013
- Automatic detection and classification of odontocete whistlesThe Journal of the Acoustical Society of America, 2013
- Automated extraction of odontocete whistle contoursThe Journal of the Acoustical Society of America, 2011
- A method for detecting whistles, moans, and other frequency contour soundsThe Journal of the Acoustical Society of America, 2011
- The evolution of the Lombard effect: 100 years of psychoacoustic researchBehaviour, 2011
- Spectrogram denoising and automated extraction of the fundamental frequency variation of dolphin whistlesThe Journal of the Acoustical Society of America, 2008
- Call detection and extraction using Bayesian inferenceApplied Acoustics, 2006
- On the communicative significance of whistles in wild killer whales ( Orcinus orca )The Science of Nature, 2002
- The relationship between behavioral activity and underwater vocalizations of the white whale, Delphinapterus leucasCanadian Journal of Zoology, 1986
- The Whistle Repertoire of the North Atlantic Pilot Whale (Globicephala melaena) and Its Relationship to Behavior and Environment1979