Animesh Garg
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Animesh Garg has a total of 177 co-authors. Here are the top co-authors by number of shared publications:
- Alán Aspuru-Guzik
Department of Chemistry, University of Toronto, Toronto, Canada
3 shared publications Ken Goldberg
3 shared publications- Kourosh Darvish
University of Toronto, Toronto, Canada
3 shared publications Kush Hari
3 shared publications- Naruki Yoshikawa
National Institute for Materials Science
3 shared publications Qinxi Yu
University of Toronto
3 shared publications
Latest Publications
- 31 December 2025
- Published by Springer Nature in Nature Computational Science
- Vol. 6 (1) , 67-82
MATTERIX: toward a digital twin for robotics-assisted chemistry laboratory automation
Accelerated materials discovery is critical for addressing global challenges. However, developing new laboratory workflows relies heavily on real-world experimental trials, and this can hinder scalability because of the need for numerous physical make-and-test iterations. Here we present MATTERIX, a multiscale, graphics processing unit-accelerated robotic simulation framework designed to create high-fidelity digital twins of chemistry laboratories, thus accelerating workflow development. This multiscale digital twin simulates robotic physical manipulation, powder and liquid dynamics, device functionalities, heat transfer and basic chemical reaction kinetics. This is enabled by integrating realistic physics simulation and photorealistic rendering with a modular graphics processing unit-accelerated semantics engine, which models logical states and continuous behaviors to simulate chemistry workflows across different levels of abstraction. MATTERIX streamlines the creation of digital twin environments through open-source asset libraries and interfaces, while enabling flexible workflow design via hierarchical plan definition and a modular skill library that incorporates learning-based methods. Our approach demonstrates sim-to-real transfer in robotic chemistry setups, reducing reliance on costly real-world experiments and enabling the testing of hypothetical automated workflows in silico.
- 31 December 2025
- Published by Springer Nature in Nature Computational Science
- Vol. 6 (1) , 67-82
MATTERIX: toward a digital twin for robotics-assisted chemistry laboratory automation
Accelerated materials discovery is critical for addressing global challenges. However, developing new laboratory workflows relies heavily on real-world experimental trials, and this can hinder scalability because of the need for numerous physical make-and-test iterations. Here we present MATTERIX, a multiscale, graphics processing unit-accelerated robotic simulation framework designed to create high-fidelity digital twins of chemistry laboratories, thus accelerating workflow development. This multiscale digital twin simulates robotic physical manipulation, powder and liquid dynamics, device functionalities, heat transfer and basic chemical reaction kinetics. This is enabled by integrating realistic physics simulation and photorealistic rendering with a modular graphics processing unit-accelerated semantics engine, which models logical states and continuous behaviors to simulate chemistry workflows across different levels of abstraction. MATTERIX streamlines the creation of digital twin environments through open-source asset libraries and interfaces, while enabling flexible workflow design via hierarchical plan definition and a modular skill library that incorporates learning-based methods. Our approach demonstrates sim-to-real transfer in robotic chemistry setups, reducing reliance on costly real-world experiments and enabling the testing of hypothetical automated workflows in silico.
- 24 September 2025
- Published by American Association for the Advancement of Science (AAAS) in Science Robotics
- Vol. 10 (106) , eadv7932
Accelerating discovery in natural science laboratories with AI and robotics: Perspectives and challenges
Science laboratory automation enables accelerated discovery in life sciences and materials. However, it requires interdisciplinary collaboration to address challenges such as robust and flexible autonomy, reproducibility, throughput, standardization, the role of human scientists, and ethics. This article highlights these issues, reflecting perspectives from leading experts in laboratory automation across different disciplines of the natural sciences.
- 24 September 2025
- Published by American Association for the Advancement of Science (AAAS) in Science Robotics
- Vol. 10 (106) , eadv7932
Accelerating discovery in natural science laboratories with AI and robotics: Perspectives and challenges
Science laboratory automation enables accelerated discovery in life sciences and materials. However, it requires interdisciplinary collaboration to address challenges such as robust and flexible autonomy, reproducibility, throughput, standardization, the role of human scientists, and ethics. This article highlights these issues, reflecting perspectives from leading experts in laboratory automation across different disciplines of the natural sciences.
- Open Access
SuFIA-BC: Generating High Quality Demonstration Data for Visuomotor Policy Learning in Surgical Subtasks
Behavior cloning facilitates the learning of dexterous manipulation skills, yet the complexity of surgical environments, the difficulty and expense of obtaining patient data, and robot calibration errors present unique challenges for surgical robot learning. We provide an enhanced surgical digital twin with photorealistic human anatomical organs, integrated into a comprehensive simulator designed to generate high-quality synthetic data to solve fundamental tasks in surgical autonomy. We present SuFIA-BC: visual Behavior Cloning policies for Surgical First Interactive Autonomy Assistants. We investigate visual observation spaces including multi-view cameras and 3D visual representations extracted from a single endoscopic camera view. Through systematic evaluation, we find that the diverse set of photorealistic surgical tasks introduced in this work enables a comprehensive evaluation of prospective behavior cloning models for the unique challenges posed by surgical environments. We observe that current state-of-the-art behavior cloning techniques struggle to solve the contact-rich and complex tasks evaluated in this work, regardless of their underlying perception or control architectures. These findings highlight the importance of customizing perception pipelines and control architectures, as well as curating larger-scale synthetic datasets that meet the specific demands of surgical tasks. Project website: https://orbit-surgical.github.io/sufia-bc/
Open AccessSuFIA-BC: Generating High Quality Demonstration Data for Visuomotor Policy Learning in Surgical Subtasks
Behavior cloning facilitates the learning of dexterous manipulation skills, yet the complexity of surgical environments, the difficulty and expense of obtaining patient data, and robot calibration errors present unique challenges for surgical robot learning. We provide an enhanced surgical digital twin with photorealistic human anatomical organs, integrated into a comprehensive simulator designed to generate high-quality synthetic data to solve fundamental tasks in surgical autonomy. We present SuFIA-BC: visual Behavior Cloning policies for Surgical First Interactive Autonomy Assistants. We investigate visual observation spaces including multi-view cameras and 3D visual representations extracted from a single endoscopic camera view. Through systematic evaluation, we find that the diverse set of photorealistic surgical tasks introduced in this work enables a comprehensive evaluation of prospective behavior cloning models for the unique challenges posed by surgical environments. We observe that current state-of-the-art behavior cloning techniques struggle to solve the contact-rich and complex tasks evaluated in this work, regardless of their underlying perception or control architectures. These findings highlight the importance of customizing perception pipelines and control architectures, as well as curating larger-scale synthetic datasets that meet the specific demands of surgical tasks. Project website: https://orbit-surgical.github.io/sufia-bc/
- Open Access
ORBIT-Surgical: An Open-Simulation Framework for Learning Surgical Augmented Dexterity
Physics-based simulations have accelerated progress in robot learning for driving, manipulation, and locomotion. Yet, a fast, accurate, and robust surgical simulation environment remains a challenge. In this paper, we present ORBIT-Surgical, a physics-based surgical robot simulation framework with photorealistic rendering in NVIDIA Omniverse. We provide 14 benchmark surgical tasks for the da Vinci Research Kit (dVRK) and Smart Tissue Autonomous Robot (STAR) which represent common subtasks in surgical training. ORBIT-Surgical leverages GPU parallelization to train reinforcement learning and imitation learning algorithms to facilitate study of robot learning to augment human surgical skills. ORBIT-Surgical also facilitates realistic synthetic data generation for active perception tasks. We demonstrate ORBIT-Surgical sim-to-real transfer of learned policies onto a physical dVRK robot. Project website: orbit-surgical.github.io
Open AccessORBIT-Surgical: An Open-Simulation Framework for Learning Surgical Augmented Dexterity
Physics-based simulations have accelerated progress in robot learning for driving, manipulation, and locomotion. Yet, a fast, accurate, and robust surgical simulation environment remains a challenge. In this paper, we present ORBIT-Surgical, a physics-based surgical robot simulation framework with photorealistic rendering in NVIDIA Omniverse. We provide 14 benchmark surgical tasks for the da Vinci Research Kit (dVRK) and Smart Tissue Autonomous Robot (STAR) which represent common subtasks in surgical training. ORBIT-Surgical leverages GPU parallelization to train reinforcement learning and imitation learning algorithms to facilitate study of robot learning to augment human surgical skills. ORBIT-Surgical also facilitates realistic synthetic data generation for active perception tasks. We demonstrate ORBIT-Surgical sim-to-real transfer of learned policies onto a physical dVRK robot. Project website: orbit-surgical.github.io
- 30 November 2023
- Published by World Scientific Pub Co Pte Ltd in Journal of Medical Robotics Research
- Vol. 08 (03n04)
Robot-Assisted Vascular Shunt Insertion with the dVRK Surgical Robot
Vascular shunt insertion is a common surgical procedure performed to restore blood flow to damaged tissues temporarily. It usually requires a surgeon and a surgical assistant. We consider three scenarios: (1) a surgeon is available locally; (2) a remote surgeon is available via teleoperation; (3) no surgeon is available. In each scenario, a minimally invasive surgical-assistant da Vinci robot operates in a different mode either by teleoperation or automation. Robotic assistance for this procedure is challenging due to precision and control uncertainty. The role of the robot in this task depends on the availability of a human surgeon. We propose a trimodal framework for vascular shunt insertion assisted by a da Vinci Research Kit (dVRK) robotic surgical assistant (RSA). To help further study for the community, we also present a physics-based simulated environment for shunt insertion built on top of the NVIDIA Isaac ORBIT simulator. We collect a large dataset of trajectories for the shunt insertion environment using ORBIT and implement these trajectories to show the simulator’s realism, showcasing the possibility for future work to use the simulator for policy learning. Physical experiments demonstrate a success rate of 65–100% for mode (1), 100% for mode (2), and 75–95% for mode (3) across vessel phantoms with different sizes, color, and material properties. For dataset and videos, see https://sites.google.com/berkeley.edu/ravsi .
- 30 November 2023
- Published by World Scientific Pub Co Pte Ltd in Journal of Medical Robotics Research
- Vol. 08 (03n04)
Robot-Assisted Vascular Shunt Insertion with the dVRK Surgical Robot
Vascular shunt insertion is a common surgical procedure performed to restore blood flow to damaged tissues temporarily. It usually requires a surgeon and a surgical assistant. We consider three scenarios: (1) a surgeon is available locally; (2) a remote surgeon is available via teleoperation; (3) no surgeon is available. In each scenario, a minimally invasive surgical-assistant da Vinci robot operates in a different mode either by teleoperation or automation. Robotic assistance for this procedure is challenging due to precision and control uncertainty. The role of the robot in this task depends on the availability of a human surgeon. We propose a trimodal framework for vascular shunt insertion assisted by a da Vinci Research Kit (dVRK) robotic surgical assistant (RSA). To help further study for the community, we also present a physics-based simulated environment for shunt insertion built on top of the NVIDIA Isaac ORBIT simulator. We collect a large dataset of trajectories for the shunt insertion environment using ORBIT and implement these trajectories to show the simulator’s realism, showcasing the possibility for future work to use the simulator for policy learning. Physical experiments demonstrate a success rate of 65–100% for mode (1), 100% for mode (2), and 75–95% for mode (3) across vessel phantoms with different sizes, color, and material properties. For dataset and videos, see https://sites.google.com/berkeley.edu/ravsi .
- Open Access
- 15 November 2023
- Published by JMIR Publications Inc. in JMIR Preprints
Responsible artificial intelligence in clinical decision support systems requires good science: Lessons learned from an international roundtable discussion (Preprint)
UNSTRUCTURED In healthcare, where increasing efficiency is essential to the demand of scale, there is immense opportunity to incorporate advances in artificial intelligence (AI). However, particularly in healthcare, these technologies must be designed to be both effective and ethical. Our objective in a multidisciplinary international roundtable discussion (Canada, United States, United Kingdom), was to identify concepts, perspectives, and considerations for AI systems in healthcare settings that are designed, developed, and deployed with good intention to empower patients and healthcare providers in a safe, trustworthy, and ethical way. We refer to this notion as responsible AI (RAI). First, we discuss the role and opportunity of AI to support collaborative healthcare (clinicians and patients working together) and increase specialist capacity. Second, we outline risks and ramifications of poorly implemented AI including bias, implications of predictors to support diagnosis, and privacy and security considerations. Third, we discuss how these risks can be mitigated through conducting “good science” by addressing biases such as representative data, probing annotation bias, and the role of the biostatistician. We also outline the need to evaluate fit for purpose through transdisciplinary collaboration to address: explainability, fairness, interpretability, transparency, as well as the role of standards, auditing, and regulatory considerations. Finally, we detail four criteria outlining determinants, considerations and rationale for developing RAI. These determinants and considerations are meant to position new AI-powered healthcare technologies primed for responsible design supporting acceptability, appropriateness, feasibility, and adoption. Future directions should expand on additional factors and monitor responsible AI implementation success to validate these criteria.
- 15 November 2023
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessResponsible artificial intelligence in clinical decision support systems requires good science: Lessons learned from an international roundtable discussion (Preprint)
UNSTRUCTURED In healthcare, where increasing efficiency is essential to the demand of scale, there is immense opportunity to incorporate advances in artificial intelligence (AI). However, particularly in healthcare, these technologies must be designed to be both effective and ethical. Our objective in a multidisciplinary international roundtable discussion (Canada, United States, United Kingdom), was to identify concepts, perspectives, and considerations for AI systems in healthcare settings that are designed, developed, and deployed with good intention to empower patients and healthcare providers in a safe, trustworthy, and ethical way. We refer to this notion as responsible AI (RAI). First, we discuss the role and opportunity of AI to support collaborative healthcare (clinicians and patients working together) and increase specialist capacity. Second, we outline risks and ramifications of poorly implemented AI including bias, implications of predictors to support diagnosis, and privacy and security considerations. Third, we discuss how these risks can be mitigated through conducting “good science” by addressing biases such as representative data, probing annotation bias, and the role of the biostatistician. We also outline the need to evaluate fit for purpose through transdisciplinary collaboration to address: explainability, fairness, interpretability, transparency, as well as the role of standards, auditing, and regulatory considerations. Finally, we detail four criteria outlining determinants, considerations and rationale for developing RAI. These determinants and considerations are meant to position new AI-powered healthcare technologies primed for responsible design supporting acceptability, appropriateness, feasibility, and adoption. Future directions should expand on additional factors and monitor responsible AI implementation success to validate these criteria.
- Open Access
- 4 October 2023
- Published by Royal Society of Chemistry (RSC) in Digital Discovery
- Vol. 2 (6) , 1745-1751
Digital pipette: open hardware for liquid transfer in self-driving laboratories
We propose an economical 3D-printed pipette, which aims to overcome the limitations of two-finger robot grippers. It enables general-purpose robot arms to achieve high precision in liquid transfer tasks that is comparable to commercial devices.
- 4 October 2023
- Published by Royal Society of Chemistry (RSC) in Digital Discovery
- Vol. 2 (6) , 1745-1751
Open AccessDigital pipette: open hardware for liquid transfer in self-driving laboratories
We propose an economical 3D-printed pipette, which aims to overcome the limitations of two-finger robot grippers. It enables general-purpose robot arms to achieve high precision in liquid transfer tasks that is comparable to commercial devices.
- Open Access
Self-Supervised Learning of Action Affordances as Interaction Modes
When humans perform a task with an articulated object, they interact with the object only in a handful of ways, while the space of all possible interactions is nearly endless. This is because humans have prior knowledge about what interactions are likely to be successful, i.e., to open a new door we first try the handle. While learning such priors without supervision is easy for humans, it is notoriously hard for machines. In this work, we tackle unsupervised learning of priors of useful interactions with articulated objects, which we call interaction modes. In contrast to the prior art, we use no supervision or privileged information; we only assume access to the depth sensor in the simulator to learn the interaction modes. More precisely, we define a successful interaction as the one changing the visual environment substantially and learn a generative model of such interactions, that can be conditioned on the desired goal state of the object. In our experiments, we show that our model covers most of the human interaction modes, outperforms existing state-of-the-art methods for affordance learning, and can generalize to objects never seen during training. Additionally, we show promising results in the goal-conditional setup, where our model can be quickly fine-tuned to perform a given task. We show in the experiments that such affordance learning predicts interaction which covers most modes of interaction for the querying articulated object and can be fine-tuned to a goal-conditional model. For supplementary: https://actaim.github.io.
Open AccessSelf-Supervised Learning of Action Affordances as Interaction Modes
When humans perform a task with an articulated object, they interact with the object only in a handful of ways, while the space of all possible interactions is nearly endless. This is because humans have prior knowledge about what interactions are likely to be successful, i.e., to open a new door we first try the handle. While learning such priors without supervision is easy for humans, it is notoriously hard for machines. In this work, we tackle unsupervised learning of priors of useful interactions with articulated objects, which we call interaction modes. In contrast to the prior art, we use no supervision or privileged information; we only assume access to the depth sensor in the simulator to learn the interaction modes. More precisely, we define a successful interaction as the one changing the visual environment substantially and learn a generative model of such interactions, that can be conditioned on the desired goal state of the object. In our experiments, we show that our model covers most of the human interaction modes, outperforms existing state-of-the-art methods for affordance learning, and can generalize to objects never seen during training. Additionally, we show promising results in the goal-conditional setup, where our model can be quickly fine-tuned to perform a given task. We show in the experiments that such affordance learning predicts interaction which covers most modes of interaction for the querying articulated object and can be fine-tuned to a goal-conditional model. For supplementary: https://actaim.github.io.
- Open Access
Learning Achievement Structure for Structured Exploration in Domains with Sparse Reward
We propose Structured Exploration with Achievements (SEA), a multi-stage reinforcement learning algorithm designed for achievement-based environments, a particular type of environment with an internal achievement set. SEA first uses offline data to learn a representation of the known achievements with a determinant loss function, then recovers the dependency graph of the learned achievements with a heuristic algorithm, and finally interacts with the environment online to learn policies that master known achievements and explore new ones with a controller built with the recovered dependency graph. We empirically demonstrate that SEA can recover the achievement structure accurately and improve exploration in hard domains such as Crafter that are procedurally generated with high-dimensional observations like images.
Open AccessLearning Achievement Structure for Structured Exploration in Domains with Sparse Reward
We propose Structured Exploration with Achievements (SEA), a multi-stage reinforcement learning algorithm designed for achievement-based environments, a particular type of environment with an internal achievement set. SEA first uses offline data to learn a representation of the known achievements with a determinant loss function, then recovers the dependency graph of the learned achievements with a heuristic algorithm, and finally interacts with the environment online to learn policies that master known achievements and explore new ones with a controller built with the recovered dependency graph. We empirically demonstrate that SEA can recover the achievement structure accurately and improve exploration in hard domains such as Crafter that are procedurally generated with high-dimensional observations like images.
- Open Access
StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos
Instructional videos are an important resource to learn procedural tasks from human demonstrations. However, the instruction steps in such videos are typically short and sparse, with most of the video being irrelevant to the procedure. This motivates the need to temporally localize the instruction steps in such videos, i.e. the task called key-step localization. Traditional methods for key-step localization require video-level human annotations and thus do not scale to large datasets. In this work, we tackle the problem with no human supervision and introduce StepFormer, a self-supervised model that discovers and localizes instruction steps in a video. StepFormer is a transformer decoder that attends to the video with learnable queries, and produces a sequence of slots capturing the key-steps in the video. We train our system on a large dataset of instructional videos, using their automatically-generated subtitles as the only source of supervision. In particular, we supervise our system with a sequence of text narrations using an order-aware loss function that filters out irrelevant phrases. We show that our model outperforms all previous unsupervised and weakly-supervised approaches on step detection and localization by a large margin on three challenging benchmarks. Moreover, our model demonstrates an emergent property to solve zero-shot multi-step localization and outperforms all relevant baselines at this task.
Open AccessStepFormer: Self-supervised Step Discovery and Localization in Instructional Videos
Instructional videos are an important resource to learn procedural tasks from human demonstrations. However, the instruction steps in such videos are typically short and sparse, with most of the video being irrelevant to the procedure. This motivates the need to temporally localize the instruction steps in such videos, i.e. the task called key-step localization. Traditional methods for key-step localization require video-level human annotations and thus do not scale to large datasets. In this work, we tackle the problem with no human supervision and introduce StepFormer, a self-supervised model that discovers and localizes instruction steps in a video. StepFormer is a transformer decoder that attends to the video with learnable queries, and produces a sequence of slots capturing the key-steps in the video. We train our system on a large dataset of instructional videos, using their automatically-generated subtitles as the only source of supervision. In particular, we supervise our system with a sequence of text narrations using an order-aware loss function that filters out irrelevant phrases. We show that our model outperforms all previous unsupervised and weakly-supervised approaches on step detection and localization by a large margin on three challenging benchmarks. Moreover, our model demonstrates an emergent property to solve zero-shot multi-step localization and outperforms all relevant baselines at this task.