Social Behavioral Phenotyping of Drosophila With a 2D–3D Hybrid CNN Framework
Authors:
Open Access
- 15 May 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 7, 67972-67982
Abstract
Behavioural phenotyping of drosphila is an important means in biological and medical research to identify genetic, pathologic or psychologic impact on animal behviour. Automated behavioural phenotyping from videos has been a desired capability that can waive long-time boring manual work in behavioral analysis. In this paper, we introduced deep learning into this challenging topic, and proposed a new 2D+3D hybrid CNN framework for drosphila’s social behavioural phenotyping. In the proposed multitask learning framework, action detection and localization of drosphila jointly is carried out with action classification, and a given video is divided into clips with fixed length. Each clip is fed into the system and a 2-D CNN is applied to extract features at frame level. Features extracted from adjacent frames are then connected and fed into a 3-D CNN with a spatial region proposal layer for classification. In such a 2D+3D hybrid framework, drosophila detection at the frame level enables the action analysis at different durations instead of a fixed period. We tested our framework with different base layers and classification architectures and validated the proposed 3D CNN based social behavioral phenotyping framework under various models, detectors and classifiers.Keywords
All Related Versions
Funding Information
- Engineering and Physical Sciences Research Council (EP/P009727/2)
This publication has 34 references indexed in Scilit:
- A Review of Human Activity Recognition MethodsFrontiers in Robotics and AI, 2015
- Deep learningNature, 2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Spatiotemporal Deformable Part Models for Action Detection2013
- JAABA: interactive machine learning for automatic annotation of animal behaviorNature Methods, 2013
- Detecting Actions, Poses, and Objects with Relational Phraselets2012
- Human Action Segmentation and Recognition Using Discriminative Semi-Markov ModelsInternational Journal of Computer Vision, 2010
- Automated home-cage behavioural phenotyping of miceNature Communications, 2010
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- Automated monitoring and analysis of social behavior in DrosophilaNature Methods, 2009