Chunxiang Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Chunxiang Wang has a total of 57 co-authors. Here are the top co-authors by number of shared publications:
- Metin Sitti
Physical Intelligence Department, Max Planck Institute for Intelligent Systems, Stuttgart 70569, Germany
8 shared publications - Chong Hong
Physical Intelligence Department, Max Planck Institute for Intelligent Systems, Stuttgart, Germany
3 shared publications - Naoki Watanabe
Institute of Low Temperature Science, Hokkaido University, Sapporo, Hokkaido 060-0819, Japan
3 shared publications - Tianlu Wang
Department of Mechanical Engineering, University of Hawaii at Manoa, Honolulu, HI, USA
3 shared publications - Yingdan Wu
State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Harbin 150001, China
3 shared publications - Zemin Liu
Physical Intelligence Department, Max Planck Institute for Intelligent Systems, Stuttgart, Germany
3 shared publications
Latest Publications
- Open Access
- 23 February 2026
- Published by Springer Nature in Nature Machine Intelligence
- Vol. 8 (2) , 276-291
Synthetic X‑ray‑driven tracking and control of miniature medical devices
The clinical translation of miniature medical devices (MMDs) for minimally invasive surgery promises transformative advances in biomedical engineering, offering enhanced precision, reduced patient trauma and faster recovery times. However, their effective deployment in complex anatomies under real-time X-ray guidance—a widely used surgical imaging modality—presents challenges such as low imaging quality and difficulties of spatial MMD control. Manual identification and operation are labour intensive and error prone. Meanwhile, deep learning-based automation is limited by the scarcity of annotated X-ray datasets of MMDs owing to costly data collection, laborious annotation and privacy constraints. Here we introduce MicroSyn-X, a framework for training computer vision models to enable robotic teleoperation of MMDs using synthesized high-fidelity, pixel-accurate, auto-labelled and domain-randomized X-ray images, eliminating manual data curation. Integrating MicroSyn-X into a teleoperated robotic system enables real-time localization and navigation of magnetic soft and magnetic liquid MMDs within both ex vivo and dynamic in vivo environments, demonstrating robustness under challenging imaging conditions of low contrast, high noise and occlusion. With these promises, we open source the X-ray MMD dataset to enable benchmarking. Addressing data scarcity and enabling real-time robotic navigation, this work advances MMD-assisted minimally invasive surgery towards next-generation precision interventions.
- 23 February 2026
- Published by Springer Nature in Nature Machine Intelligence
- Vol. 8 (2) , 276-291
Open AccessSynthetic X‑ray‑driven tracking and control of miniature medical devices
The clinical translation of miniature medical devices (MMDs) for minimally invasive surgery promises transformative advances in biomedical engineering, offering enhanced precision, reduced patient trauma and faster recovery times. However, their effective deployment in complex anatomies under real-time X-ray guidance—a widely used surgical imaging modality—presents challenges such as low imaging quality and difficulties of spatial MMD control. Manual identification and operation are labour intensive and error prone. Meanwhile, deep learning-based automation is limited by the scarcity of annotated X-ray datasets of MMDs owing to costly data collection, laborious annotation and privacy constraints. Here we introduce MicroSyn-X, a framework for training computer vision models to enable robotic teleoperation of MMDs using synthesized high-fidelity, pixel-accurate, auto-labelled and domain-randomized X-ray images, eliminating manual data curation. Integrating MicroSyn-X into a teleoperated robotic system enables real-time localization and navigation of magnetic soft and magnetic liquid MMDs within both ex vivo and dynamic in vivo environments, demonstrating robustness under challenging imaging conditions of low contrast, high noise and occlusion. With these promises, we open source the X-ray MMD dataset to enable benchmarking. Addressing data scarcity and enabling real-time robotic navigation, this work advances MMD-assisted minimally invasive surgery towards next-generation precision interventions.
- Top Cited PapersOpen Access
- 14 January 2026
- Published by Springer Nature in Nature
- Vol. 649 (8098) , 885-893
3D-printed low-voltage-driven ciliary hydrogel microactuators
Micrometre-sized, densely packed natural cilia that perform non-reciprocal 3D motions with dynamically tunable collective patterns are crucial for biological processes such as microscale locomotion 1 , nutrient acquisition 2 , cell trafficking 3–5 and embryonic and neurological development 6–8 . However, replicating these motions in artificial systems remains challenging given the limits of scalable, locally controllable soft-bodied actuation at the micrometre scale. Overcoming this challenge would enhance our understanding of ciliary dynamics, clarify their biological importance and enable new microscale devices and bioinspired technologies. Here we show a previously unrecognized fast electrical response of micrometre-scale hydrogels, induced by voltages down to 1.5 V without hydrolysis, with bending motions driven by ion migration across a nanometre-scale hydrogel network 3D-printed by two-photon polymerization, occurring within milliseconds. On the basis of these findings, we print gel microcilia arrays composed of a soft acrylic acid-co-acrylamide (AAc-co-AAm) hydrogel (modulus of approximately 1,000 Pa) that respond to electrical stimuli within milliseconds. Each microcilium measures 2–10 µm in diameter and 18–90 µm in height, achieving 3D rotational bending motion at up to 40 Hz, mirroring the geometry and dynamics of natural cilia. These gel microcilia maintain functionality after 330,000 continuous actuation cycles with less than 30% performance degradation. The gel microcilia arrays can be integrated on flexible polyimide substrates and fabricated at large scale using conventional lithography techniques. They also offer individual dynamic control by means of microelectrode arrays and enable fluid manipulation and particle transport at the micrometre scale.
- 14 January 2026
- Published by Springer Nature in Nature
- Vol. 649 (8098) , 885-893
Top Cited PapersOpen Access3D-printed low-voltage-driven ciliary hydrogel microactuators
Micrometre-sized, densely packed natural cilia that perform non-reciprocal 3D motions with dynamically tunable collective patterns are crucial for biological processes such as microscale locomotion 1 , nutrient acquisition 2 , cell trafficking 3–5 and embryonic and neurological development 6–8 . However, replicating these motions in artificial systems remains challenging given the limits of scalable, locally controllable soft-bodied actuation at the micrometre scale. Overcoming this challenge would enhance our understanding of ciliary dynamics, clarify their biological importance and enable new microscale devices and bioinspired technologies. Here we show a previously unrecognized fast electrical response of micrometre-scale hydrogels, induced by voltages down to 1.5 V without hydrolysis, with bending motions driven by ion migration across a nanometre-scale hydrogel network 3D-printed by two-photon polymerization, occurring within milliseconds. On the basis of these findings, we print gel microcilia arrays composed of a soft acrylic acid-co-acrylamide (AAc-co-AAm) hydrogel (modulus of approximately 1,000 Pa) that respond to electrical stimuli within milliseconds. Each microcilium measures 2–10 µm in diameter and 18–90 µm in height, achieving 3D rotational bending motion at up to 40 Hz, mirroring the geometry and dynamics of natural cilia. These gel microcilia maintain functionality after 330,000 continuous actuation cycles with less than 30% performance degradation. The gel microcilia arrays can be integrated on flexible polyimide substrates and fabricated at large scale using conventional lithography techniques. They also offer individual dynamic control by means of microelectrode arrays and enable fluid manipulation and particle transport at the micrometre scale.
- Open Access
- 2 January 2026
- Published by American Association for the Advancement of Science (AAAS) in Science Advances
- Vol. 12 (1) , eadx9616
In situ mechanostimulation of biohybrid millirobots for enhanced cell functionality and delivery
This study proposes a perforated, soft millirobot with dual functions: in situ mechanostimulation to enhance cell functionality and local cell delivery. Following protein modification and silica coating, the soft millirobots exhibit excellent biocompatibility, promoting cell adhesion and tissue ingrowth within their perforated architectures under both in vitro and in vivo conditions. They can apply in situ mechanostimulation to various cellular morphologies, including two-dimensional (2D) cell sheets, 3D cell–laden hydrogels, and ex vivo tissue models. The mechanical stimulation improves the functionality of muscle cells by enhancing cellular orientation, myotube contraction, and myocyte differentiation. In parallel, we develop an integrated robotic platform combining magnetic actuation with ultrasound imaging. It demonstrates the proof of principle that delivers 2D cell-sheet and 3D cell–laden biohybrid millirobots to narrow regions in an ex vivo pig liver model. This work expands the potential applications of soft millirobots in mechanobiology studies and future cell-based therapies.
- 2 January 2026
- Published by American Association for the Advancement of Science (AAAS) in Science Advances
- Vol. 12 (1) , eadx9616
Open AccessIn situ mechanostimulation of biohybrid millirobots for enhanced cell functionality and delivery
This study proposes a perforated, soft millirobot with dual functions: in situ mechanostimulation to enhance cell functionality and local cell delivery. Following protein modification and silica coating, the soft millirobots exhibit excellent biocompatibility, promoting cell adhesion and tissue ingrowth within their perforated architectures under both in vitro and in vivo conditions. They can apply in situ mechanostimulation to various cellular morphologies, including two-dimensional (2D) cell sheets, 3D cell–laden hydrogels, and ex vivo tissue models. The mechanical stimulation improves the functionality of muscle cells by enhancing cellular orientation, myotube contraction, and myocyte differentiation. In parallel, we develop an integrated robotic platform combining magnetic actuation with ultrasound imaging. It demonstrates the proof of principle that delivers 2D cell-sheet and 3D cell–laden biohybrid millirobots to narrow regions in an ex vivo pig liver model. This work expands the potential applications of soft millirobots in mechanobiology studies and future cell-based therapies.
- Open Access
- 16 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 30 (6) , 7717-7727
Synthetic Data-Assisted Miniature Medical Robot Navigation via Ultrasound Imaging
Wireless miniature robots are promising for minimally invasive biomedical applications. Effective tracking and navigation are essential for their safe deployment, but challenges persist in medical imaging and robot control, especially in localizing the robot in complex imaging scenes. Deep learning, though powerful for object identification, requires large supervised datasets, limiting its clinical applications due to the difficulty and cost of acquiring realistic data. Furthermore, miniature robots frequently exit the field of view of imaging systems, hindering continuous observation. Here, we present a framework for real-time magnetic navigation of wireless miniature robots using ultrasound imaging, leveraging synthetic data generation for deep learning-based detection. First, artificially generated synthetic data is combined with real data from synthetic materials to train a neural network capable of detecting versatile robots in real tissues. Then, a robotic system is developed to automatically track the robot with an ultrasound probe during magnetic actuation in tortuous lumens. With 85% less human-labeled data within synthetic materials, our approach effectively detects versatile robots in ex-vivo tissues, reducing data scarcity, imbalance, and manual labeling burdens. Demonstrations of automatic robot navigation through tortuous lumens in complex ultrasound scenes validate its effectiveness, enhancing the safe applicability of miniature medical robots in complex environments.
- 16 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 30 (6) , 7717-7727
Open AccessSynthetic Data-Assisted Miniature Medical Robot Navigation via Ultrasound Imaging
Wireless miniature robots are promising for minimally invasive biomedical applications. Effective tracking and navigation are essential for their safe deployment, but challenges persist in medical imaging and robot control, especially in localizing the robot in complex imaging scenes. Deep learning, though powerful for object identification, requires large supervised datasets, limiting its clinical applications due to the difficulty and cost of acquiring realistic data. Furthermore, miniature robots frequently exit the field of view of imaging systems, hindering continuous observation. Here, we present a framework for real-time magnetic navigation of wireless miniature robots using ultrasound imaging, leveraging synthetic data generation for deep learning-based detection. First, artificially generated synthetic data is combined with real data from synthetic materials to train a neural network capable of detecting versatile robots in real tissues. Then, a robotic system is developed to automatically track the robot with an ultrasound probe during magnetic actuation in tortuous lumens. With 85% less human-labeled data within synthetic materials, our approach effectively detects versatile robots in ex-vivo tissues, reducing data scarcity, imbalance, and manual labeling burdens. Demonstrations of automatic robot navigation through tortuous lumens in complex ultrasound scenes validate its effectiveness, enhancing the safe applicability of miniature medical robots in complex environments.
- Open Access
- 8 November 2024
- Published by American Association for the Advancement of Science (AAAS) in Science Advances
- Vol. 10 (45) , eadq1951
Heterogeneous multiple soft millirobots in three-dimensional lumens
Miniature soft robots offer opportunities for safe and physically adaptive medical interventions in hard-to-reach regions. Deploying multiple robots could further enhance the efficacy and multifunctionality of these operations. However, multirobot deployment in physiologically relevant three-dimensional (3D) tubular structures is limited by the lack of effective mechanisms for independent control of miniature magnetic soft robots. This work presents a framework leveraging the shape-adaptive robotic design and heterogeneous resistance from robot-lumen interactions to enable magnetic multirobot control. We first compute influence and actuation regions to quantify robot movement. Subsequently, a path planning algorithm generates the trajectory of a permanent magnet for multirobot navigation in 3D lumens. Last, robots are controlled individually in multilayer lumen networks under medical imaging. Demonstrations of multilocation cargo delivery and flow diversion manifest their potential to enhance biomedical functions. This framework offers a solution to multirobot actuation benefiting applications across different miniature robotic devices in complex environments.
- 8 November 2024
- Published by American Association for the Advancement of Science (AAAS) in Science Advances
- Vol. 10 (45) , eadq1951
Open AccessHeterogeneous multiple soft millirobots in three-dimensional lumens
Miniature soft robots offer opportunities for safe and physically adaptive medical interventions in hard-to-reach regions. Deploying multiple robots could further enhance the efficacy and multifunctionality of these operations. However, multirobot deployment in physiologically relevant three-dimensional (3D) tubular structures is limited by the lack of effective mechanisms for independent control of miniature magnetic soft robots. This work presents a framework leveraging the shape-adaptive robotic design and heterogeneous resistance from robot-lumen interactions to enable magnetic multirobot control. We first compute influence and actuation regions to quantify robot movement. Subsequently, a path planning algorithm generates the trajectory of a permanent magnet for multirobot navigation in 3D lumens. Last, robots are controlled individually in multilayer lumen networks under medical imaging. Demonstrations of multilocation cargo delivery and flow diversion manifest their potential to enhance biomedical functions. This framework offers a solution to multirobot actuation benefiting applications across different miniature robotic devices in complex environments.
- 13 March 2024
- Published by American Association for the Advancement of Science (AAAS) in Science Robotics
- Vol. 9 (88) , eadi5155
Wireless flow-powered miniature robot capable of traversing tubular structures
Wireless millimeter-scale robots capable of navigating through fluid-flowing tubular structures hold substantial potential for inspection, maintenance, or repair use in nuclear, industrial, and medical applications. However, prevalent reliance on external powering constrains these robots’ operational range and applicable environments. Alternatives with onboard powering must trade off size, functionality, and operation duration. Here, we propose a wireless millimeter-scale wheeled robot capable of using environmental flows to power and actuate its long-distance locomotion through complex pipelines. The flow-powering module can convert flow energy into mechanical energy, achieving an impeller speed of up to 9595 revolutions per minute, accompanied by an output power density of 11.7 watts per cubic meter and an efficiency of 33.7%. A miniature gearbox module can further transmit the converted mechanical energy into the robot’s locomotion system, allowing the robot to move against water flow at an average rate of up to 1.05 meters per second. The robot’s motion status (moving against/with flow or pausing) can be switched using an external magnetic field or an onboard mechanical regulator, contingent on different proposed control designs. In addition, we designed kirigami-based soft wheels for adaptive locomotion. The robot can move against flows of various substances within pipes featuring complex geometries and diverse materials. Solely powered by flow, the robot can transport cylindrical payloads with a diameter of up to 55% of the pipe’s diameter and carry devices such as an endoscopic camera for pipeline inspection, a wireless temperature sensor for environmental temperature monitoring, and a leak-stopper shell for infrastructure maintenance.
- 13 March 2024
- Published by American Association for the Advancement of Science (AAAS) in Science Robotics
- Vol. 9 (88) , eadi5155
Wireless flow-powered miniature robot capable of traversing tubular structures
Wireless millimeter-scale robots capable of navigating through fluid-flowing tubular structures hold substantial potential for inspection, maintenance, or repair use in nuclear, industrial, and medical applications. However, prevalent reliance on external powering constrains these robots’ operational range and applicable environments. Alternatives with onboard powering must trade off size, functionality, and operation duration. Here, we propose a wireless millimeter-scale wheeled robot capable of using environmental flows to power and actuate its long-distance locomotion through complex pipelines. The flow-powering module can convert flow energy into mechanical energy, achieving an impeller speed of up to 9595 revolutions per minute, accompanied by an output power density of 11.7 watts per cubic meter and an efficiency of 33.7%. A miniature gearbox module can further transmit the converted mechanical energy into the robot’s locomotion system, allowing the robot to move against water flow at an average rate of up to 1.05 meters per second. The robot’s motion status (moving against/with flow or pausing) can be switched using an external magnetic field or an onboard mechanical regulator, contingent on different proposed control designs. In addition, we designed kirigami-based soft wheels for adaptive locomotion. The robot can move against flows of various substances within pipes featuring complex geometries and diverse materials. Solely powered by flow, the robot can transport cylindrical payloads with a diameter of up to 55% of the pipe’s diameter and carry devices such as an endoscopic camera for pipeline inspection, a wireless temperature sensor for environmental temperature monitoring, and a leak-stopper shell for infrastructure maintenance.
- Open Access
- 9 June 2023
- Published by American Association for the Advancement of Science (AAAS) in Science Advances
- Vol. 9 (23) , eadg3988
In situ sensing physiological properties of biological tissues using wireless miniature soft robots
Implanted electronic sensors, compared with conventional medical imaging, allow monitoring of advanced physiological properties of soft biological tissues continuously, such as adhesion, pH, viscoelasticity, and biomarkers for disease diagnosis. However, they are typically invasive, requiring being deployed by surgery, and frequently cause inflammation. Here we propose a minimally invasive method of using wireless miniature soft robots to in situ sense the physiological properties of tissues. By controlling robot-tissue interaction using external magnetic fields, visualized by medical imaging, we can recover tissue properties precisely from the robot shape and magnetic fields. We demonstrate that the robot can traverse tissues with multimodal locomotion and sense the adhesion, pH, and viscoelasticity on porcine and mice gastrointestinal tissues ex vivo, tracked by x-ray or ultrasound imaging. With the unprecedented capability of sensing tissue physiological properties with minimal invasion and high resolution deep inside our body, this technology can potentially enable critical applications in both basic research and clinical practice.
- 9 June 2023
- Published by American Association for the Advancement of Science (AAAS) in Science Advances
- Vol. 9 (23) , eadg3988
Open AccessIn situ sensing physiological properties of biological tissues using wireless miniature soft robots
Implanted electronic sensors, compared with conventional medical imaging, allow monitoring of advanced physiological properties of soft biological tissues continuously, such as adhesion, pH, viscoelasticity, and biomarkers for disease diagnosis. However, they are typically invasive, requiring being deployed by surgery, and frequently cause inflammation. Here we propose a minimally invasive method of using wireless miniature soft robots to in situ sense the physiological properties of tissues. By controlling robot-tissue interaction using external magnetic fields, visualized by medical imaging, we can recover tissue properties precisely from the robot shape and magnetic fields. We demonstrate that the robot can traverse tissues with multimodal locomotion and sense the adhesion, pH, and viscoelasticity on porcine and mice gastrointestinal tissues ex vivo, tracked by x-ray or ultrasound imaging. With the unprecedented capability of sensing tissue physiological properties with minimal invasion and high resolution deep inside our body, this technology can potentially enable critical applications in both basic research and clinical practice.
- 13 September 2022
- Published by Springer Nature in Journal of Materials Science: Materials in Electronics
- Vol. 33 (28) , 22560-22572
Synthesis of centimeter-scale WS2 membrane by chemical vapor deposition
- 13 September 2022
- Published by Springer Nature in Journal of Materials Science: Materials in Electronics
- Vol. 33 (28) , 22560-22572
Synthesis of centimeter-scale WS2 membrane by chemical vapor deposition
- Top Cited PapersOpen Access
- 27 May 2022
- Published by American Association for the Advancement of Science (AAAS) in Science Advances
- Vol. 8 (21) , eabn3431
Wireless soft millirobots for climbing three-dimensional surfaces in confined spaces
Wireless soft-bodied robots at the millimeter scale allow traversing very confined unstructured terrains with minimal invasion and safely interacting with the surrounding environment. However, existing untethered soft millirobots still lack the ability of climbing, reversible controlled surface adhesion, and long-term retention on unstructured three-dimensional (3D) surfaces, limiting their use in biomedical and environmental applications. Here, we report a fundamental peeling-and-loading mechanism to allow untethered soft-bodied robots to climb 3D surfaces by using both the soft-body deformation and whole-body motion of the robot under external magnetic fields. This generic mechanism is implemented with different adhesive robot footpad designs, allowing vertical and inverted surface climbing on diverse 3D surfaces with complex geometries and different surface properties. With the unique robot footpad designs that integrate microstructured adhesives and tough bioadhesives, the soft climbing robot could achieve controllable adhesion and friction to climb 3D soft and wet surfaces including porcine tissues, which paves the way for future environmental inspection and minimally invasive medicine applications.
- 27 May 2022
- Published by American Association for the Advancement of Science (AAAS) in Science Advances
- Vol. 8 (21) , eabn3431
Top Cited PapersOpen AccessWireless soft millirobots for climbing three-dimensional surfaces in confined spaces
Wireless soft-bodied robots at the millimeter scale allow traversing very confined unstructured terrains with minimal invasion and safely interacting with the surrounding environment. However, existing untethered soft millirobots still lack the ability of climbing, reversible controlled surface adhesion, and long-term retention on unstructured three-dimensional (3D) surfaces, limiting their use in biomedical and environmental applications. Here, we report a fundamental peeling-and-loading mechanism to allow untethered soft-bodied robots to climb 3D surfaces by using both the soft-body deformation and whole-body motion of the robot under external magnetic fields. This generic mechanism is implemented with different adhesive robot footpad designs, allowing vertical and inverted surface climbing on diverse 3D surfaces with complex geometries and different surface properties. With the unique robot footpad designs that integrate microstructured adhesives and tough bioadhesives, the soft climbing robot could achieve controllable adhesion and friction to climb 3D soft and wet surfaces including porcine tissues, which paves the way for future environmental inspection and minimally invasive medicine applications.
- 26 January 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 19 (2) , 1058-1069
DanioSense: Automated High-Throughput Quantification of Zebrafish Larvae Group Movement
The capability to obtain detailed motility information of model organisms is fundamental to reveal their functional and social behavior characteristics. Zebrafish is a powerful vertebrate model organism. Despite recent success in the automatic quantification of adult zebrafish movement, it remains a laborious task for group zebrafish larval tracking due to their similar appearance, frequent occlusions, and highly discontinuous kinematics. This article presents DanioSense (DS), an automatic tracker for group larval zebrafish, to overcome these tracking challenges. The integration of a light convolutional neural network and a centerline extraction algorithm enables the tracker to localize individuals even in occlusion cases where objects’ identities are prone to switch. With reliable detections, an adaptive Kalman filter is designed to optimally estimate locomotive parameters, which is also used for object reidentification accomplished by a two-stage data association protocol. Experimental results demonstrated a tracking accuracy of over 97%, median errors of $102~{\mathrm{\mu m}}$ , and 8.8° for the position and orientation measurement, and a processing speed of over 30 frames/s with a normal computer configuration. DS provides detailed quantitative data for a large-scale larvae group in nearly real time, highly boosting the efficiency of characterizing individual phenotypes and analyzing social interactions. Note to Practitioners —This article aimed to tackle the problem of automated tracking groups of zebrafish larvae, an ideal vertebrate model organism for large-scale chemical and genetic screens. The task of group tracking is to record each individual’s movement and calculate their position, velocity, direction, and other parameters for further analysis, where the correct identity of each individual must be maintained. Existing algorithms either switch larvae’ identities easily or are unable to achieve online tracking due to the limitations of their methods to address individuals’ intersections. DanioSense (DS) adopts a convolutional neural network to identify larval heads whenever they intersect and uses an adaptive Kalman filter to calculate the movement parameters optimally. Besides, a range of visualization options is designed to bring insight into underlying patterns through massive amounts of data. Theoretically, this algorithm’s approach to solving intersections and calculating movement statistics can also apply to other fish-like animals. Its visualization options are applicable to other tracking systems. The key advantage of Daniosense over existing trackers is the capability to track each larva within a group and output detailed quantitative data in nearly real time. The tracking performance of DS is based on the quality of image segmentation and the success rate of classifying samples. Many state-of-the-art image segmentation and classification neural networks can be adopted to extend this system’s applications to more complex environments but at a higher computation and time cost, which is a tradeoff between efficiency and capability. Some applications require a higher video sampling rate, so the system’s processing speed needs to be further improved with better hardware and software framework optimization. The next steps include improving the processing efficiency, providing more tracking modules and visualization options, and extending its application fields.
- 26 January 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 19 (2) , 1058-1069
DanioSense: Automated High-Throughput Quantification of Zebrafish Larvae Group Movement
The capability to obtain detailed motility information of model organisms is fundamental to reveal their functional and social behavior characteristics. Zebrafish is a powerful vertebrate model organism. Despite recent success in the automatic quantification of adult zebrafish movement, it remains a laborious task for group zebrafish larval tracking due to their similar appearance, frequent occlusions, and highly discontinuous kinematics. This article presents DanioSense (DS), an automatic tracker for group larval zebrafish, to overcome these tracking challenges. The integration of a light convolutional neural network and a centerline extraction algorithm enables the tracker to localize individuals even in occlusion cases where objects’ identities are prone to switch. With reliable detections, an adaptive Kalman filter is designed to optimally estimate locomotive parameters, which is also used for object reidentification accomplished by a two-stage data association protocol. Experimental results demonstrated a tracking accuracy of over 97%, median errors of $102~{\mathrm{\mu m}}$ , and 8.8° for the position and orientation measurement, and a processing speed of over 30 frames/s with a normal computer configuration. DS provides detailed quantitative data for a large-scale larvae group in nearly real time, highly boosting the efficiency of characterizing individual phenotypes and analyzing social interactions. Note to Practitioners —This article aimed to tackle the problem of automated tracking groups of zebrafish larvae, an ideal vertebrate model organism for large-scale chemical and genetic screens. The task of group tracking is to record each individual’s movement and calculate their position, velocity, direction, and other parameters for further analysis, where the correct identity of each individual must be maintained. Existing algorithms either switch larvae’ identities easily or are unable to achieve online tracking due to the limitations of their methods to address individuals’ intersections. DanioSense (DS) adopts a convolutional neural network to identify larval heads whenever they intersect and uses an adaptive Kalman filter to calculate the movement parameters optimally. Besides, a range of visualization options is designed to bring insight into underlying patterns through massive amounts of data. Theoretically, this algorithm’s approach to solving intersections and calculating movement statistics can also apply to other fish-like animals. Its visualization options are applicable to other tracking systems. The key advantage of Daniosense over existing trackers is the capability to track each larva within a group and output detailed quantitative data in nearly real time. The tracking performance of DS is based on the quality of image segmentation and the success rate of classifying samples. Many state-of-the-art image segmentation and classification neural networks can be adopted to extend this system’s applications to more complex environments but at a higher computation and time cost, which is a tradeoff between efficiency and capability. Some applications require a higher video sampling rate, so the system’s processing speed needs to be further improved with better hardware and software framework optimization. The next steps include improving the processing efficiency, providing more tracking modules and visualization options, and extending its application fields.