Gregory D. Hager
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Gregory D. Hager has a total of 61 co-authors. Here are the top co-authors by number of shared publications:
- Masaru Ishii
Department of Otolaryngology—Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA
5 shared publications - Russell H. Taylor
Department of Computer Science, Johns Hopkins University
4 shared publications Daniel Mirota
The Johns Hopkins Univ. (United States)
2 shared publications- Felix Jonathan
Department of Computer Science Johns Hopkins Uni- versity 3400 N Charles Street 21218 Baltimore MD USA
2 shared publications Jin Bai
Johns Hopkins University 3400 N. Charles Street 21218 Baltimore Maryland
2 shared publications- Jonathan D. Jones
Department of Electrical Engineering, Johns Hopkins University, Baltimore, MD, USA
2 shared publications
Latest Publications
- 24 April 2025
- Published by Springer Nature in International Journal of Computer Assisted Radiology and Surgery
- Vol. 20 (10) , 2087-2093
The quiet eye phenomenon in minimally invasive surgery
Purpose The quiet eye (QE) behavior is a gaze behavior that has been extensively studied in sports training and has been associated with higher level of expertise in multiple sports. In this paper, we report our observations of this gaze behavior in two minimally invasive surgery settings and we report how this behavior changes based on task success and the surgeon’s expertise level. Methods We investigated the QE behavior in two independently collected data sets in a sinus surgery setting and a robotic surgery setting. The sinus surgery data set was used to study how the QE behavior changes in successful and unsuccessful tasks. The robotic surgery data set was used to study how the QE behavior changes based on the surgeon’s expertise level. Results Using the sinus surgery data set, our results show that the QE behavior is more likely to occur and that its duration is significantly longer, in successful tasks, compared with unsuccessful ones. Using the robotic surgery data set, our results show similar trends in tasks performed by experienced surgeons, compared with less experienced ones. Conclusion The results of our study open the door to use the QE behavior in training and skill assessment in the explored minimally invasive surgery settings. Training novices to adopt the QE behavior can potentially improve their motor skill learning, replicating the success of doing so in sports training. In addition, the well-defined characteristics of the QE behavior can provide an explainable way to distinguish between different skill levels in minimally invasive surgery.
- 24 April 2025
- Published by Springer Nature in International Journal of Computer Assisted Radiology and Surgery
- Vol. 20 (10) , 2087-2093
The quiet eye phenomenon in minimally invasive surgery
Purpose The quiet eye (QE) behavior is a gaze behavior that has been extensively studied in sports training and has been associated with higher level of expertise in multiple sports. In this paper, we report our observations of this gaze behavior in two minimally invasive surgery settings and we report how this behavior changes based on task success and the surgeon’s expertise level. Methods We investigated the QE behavior in two independently collected data sets in a sinus surgery setting and a robotic surgery setting. The sinus surgery data set was used to study how the QE behavior changes in successful and unsuccessful tasks. The robotic surgery data set was used to study how the QE behavior changes based on the surgeon’s expertise level. Results Using the sinus surgery data set, our results show that the QE behavior is more likely to occur and that its duration is significantly longer, in successful tasks, compared with unsuccessful ones. Using the robotic surgery data set, our results show similar trends in tasks performed by experienced surgeons, compared with less experienced ones. Conclusion The results of our study open the door to use the QE behavior in training and skill assessment in the explored minimally invasive surgery settings. Training novices to adopt the QE behavior can potentially improve their motor skill learning, replicating the success of doing so in sports training. In addition, the well-defined characteristics of the QE behavior can provide an explainable way to distinguish between different skill levels in minimally invasive surgery.
- Open Access
A Quantitative Evaluation of Dense 3D Reconstruction of Sinus Anatomy from Monocular Endoscopic Video
Generating accurate 3D reconstructions from endoscopic video is a promising avenue for longitudinal radiation-free analysis of sinus anatomy and surgical outcomes. Several methods for monocular reconstruction have been proposed, yielding visually pleasant 3D anatomical structures by retrieving relative camera poses with structure-from-motion-type algorithms and fusion of monocular depth estimates. However, due to the complex properties of the underlying algorithms and endoscopic scenes, the reconstruction pipeline may perform poorly or fail unexpectedly. Further, acquiring medical data conveys additional challenges, presenting difficulties in quantitatively benchmarking these models, understanding failure cases, and identifying critical components that contribute to their precision. In this work, we perform a quantitative analysis of a self-supervised approach for sinus reconstruction using endoscopic sequences paired with optical tracking and high-resolution computed tomography acquired from nine ex-vivo specimens. Our results show that the generated reconstructions are in high agreement with the anatomy, yielding an average point-to-mesh error of 0.91 mm between reconstructions and CT segmentations. However, in a point-to-point matching scenario, relevant for endoscope tracking and navigation, we found average target registration errors of 6.58 mm. We identified that pose and depth estimation inaccuracies contribute equally to this error and that locally consistent sequences with shorter trajectories generate more accurate reconstructions. These results suggest that achieving global consistency between relative camera poses and estimated depths with the anatomy is essential. In doing so, we can ensure proper synergy between all components of the pipeline for improved reconstructions that will facilitate clinical application of this innovative technology.
Open AccessA Quantitative Evaluation of Dense 3D Reconstruction of Sinus Anatomy from Monocular Endoscopic Video
Generating accurate 3D reconstructions from endoscopic video is a promising avenue for longitudinal radiation-free analysis of sinus anatomy and surgical outcomes. Several methods for monocular reconstruction have been proposed, yielding visually pleasant 3D anatomical structures by retrieving relative camera poses with structure-from-motion-type algorithms and fusion of monocular depth estimates. However, due to the complex properties of the underlying algorithms and endoscopic scenes, the reconstruction pipeline may perform poorly or fail unexpectedly. Further, acquiring medical data conveys additional challenges, presenting difficulties in quantitatively benchmarking these models, understanding failure cases, and identifying critical components that contribute to their precision. In this work, we perform a quantitative analysis of a self-supervised approach for sinus reconstruction using endoscopic sequences paired with optical tracking and high-resolution computed tomography acquired from nine ex-vivo specimens. Our results show that the generated reconstructions are in high agreement with the anatomy, yielding an average point-to-mesh error of 0.91 mm between reconstructions and CT segmentations. However, in a point-to-point matching scenario, relevant for endoscope tracking and navigation, we found average target registration errors of 6.58 mm. We identified that pose and depth estimation inaccuracies contribute equally to this error and that locally consistent sequences with shorter trajectories generate more accurate reconstructions. These results suggest that achieving global consistency between relative camera poses and estimated depths with the anatomy is essential. In doing so, we can ensure proper synergy between all components of the pipeline for improved reconstructions that will facilitate clinical application of this innovative technology.
- Open Access
DASZL: Dynamic Action Signatures for Zero-shot Learning
There are many realistic applications of activity recognition where the set of potential activity descriptions is combinatorially large. This makes end-to-end supervised training of a recognition system impractical as no training set is practically able to encompass the entire label set. In this paper, we present an approach to fine-grained recognition that models activities as compositions of dynamic action signatures. This compositional approach allows us to reframe fine-grained recognition as zero-shot activity recognition, where a detector is composed "on the fly" from simple first-principles state machines supported by deep-learned components. We evaluate our method on the Olympic Sports and UCF101 datasets, where our model establishes a new state of the art under multiple experimental paradigms. We also extend this method to form a unique framework for zero-shot joint segmentation and classification of activities in video and demonstrate the first results in zero-shot decoding of complex action sequences on a widely-used surgical dataset. Lastly, we show that we can use off-the-shelf object detectors to recognize activities in completely de-novo settings with no additional training.
Open AccessDASZL: Dynamic Action Signatures for Zero-shot Learning
There are many realistic applications of activity recognition where the set of potential activity descriptions is combinatorially large. This makes end-to-end supervised training of a recognition system impractical as no training set is practically able to encompass the entire label set. In this paper, we present an approach to fine-grained recognition that models activities as compositions of dynamic action signatures. This compositional approach allows us to reframe fine-grained recognition as zero-shot activity recognition, where a detector is composed "on the fly" from simple first-principles state machines supported by deep-learned components. We evaluate our method on the Olympic Sports and UCF101 datasets, where our model establishes a new state of the art under multiple experimental paradigms. We also extend this method to form a unique framework for zero-shot joint segmentation and classification of activities in video and demonstrate the first results in zero-shot decoding of complex action sequences on a widely-used surgical dataset. Lastly, we show that we can use off-the-shelf object detectors to recognize activities in completely de-novo settings with no additional training.
- Open Access
Artificial Intelligence-based Clinical Decision Support for COVID-19 -- Where Art Thou?
The COVID-19 crisis has brought about new clinical questions, new workflows, and accelerated distributed healthcare needs. While artificial intelligence (AI)-based clinical decision support seemed to have matured, the application of AI-based tools for COVID-19 has been limited to date. In this perspective piece, we identify opportunities and requirements for AI-based clinical decision support systems and highlight challenges that impact "AI readiness" for rapidly emergent healthcare challenges.
Open AccessArtificial Intelligence-based Clinical Decision Support for COVID-19 -- Where Art Thou?
The COVID-19 crisis has brought about new clinical questions, new workflows, and accelerated distributed healthcare needs. While artificial intelligence (AI)-based clinical decision support seemed to have matured, the application of AI-based tools for COVID-19 has been limited to date. In this perspective piece, we identify opportunities and requirements for AI-based clinical decision support systems and highlight challenges that impact "AI readiness" for rapidly emergent healthcare challenges.
- Open Access
A Unified Framework for Multi-View Multi-Class Object Pose Estimation
One core challenge in object pose estimation is to ensure accurate and robust performance for large numbers of diverse foreground objects amidst complex background clutter. In this work, we present a scalable framework for accurately inferring six Degree-of-Freedom (6-DoF) pose for a large number of object classes from single or multiple views. To learn discriminative pose features, we integrate three new capabilities into a deep Convolutional Neural Network (CNN): an inference scheme that combines both classification and pose regression based on a uniform tessellation of the Special Euclidean group in three dimensions (SE(3)), the fusion of class priors into the training process via a tiled class map, and an additional regularization using deep supervision with an object mask. Further, an efficient multi-view framework is formulated to address single-view ambiguity. We show that this framework consistently improves the performance of the single-view network. We evaluate our method on three large-scale benchmarks: YCB-Video, JHUScene-50 and ObjectNet-3D. Our approach achieves competitive or superior performance over the current state-of-the-art methods.
Open AccessA Unified Framework for Multi-View Multi-Class Object Pose Estimation
One core challenge in object pose estimation is to ensure accurate and robust performance for large numbers of diverse foreground objects amidst complex background clutter. In this work, we present a scalable framework for accurately inferring six Degree-of-Freedom (6-DoF) pose for a large number of object classes from single or multiple views. To learn discriminative pose features, we integrate three new capabilities into a deep Convolutional Neural Network (CNN): an inference scheme that combines both classification and pose regression based on a uniform tessellation of the Special Euclidean group in three dimensions (SE(3)), the fusion of class priors into the training process via a tiled class map, and an additional regularization using deep supervision with an object mask. Further, an efficient multi-view framework is formulated to address single-view ambiguity. We show that this framework consistently improves the performance of the single-view network. We evaluate our method on three large-scale benchmarks: YCB-Video, JHUScene-50 and ObjectNet-3D. Our approach achieves competitive or superior performance over the current state-of-the-art methods.
- Open Access
Adversarial Deep Structured Nets for Mass Segmentation from Mammograms
Mass segmentation provides effective morphological features which are important for mass diagnosis. In this work, we propose a novel end-to-end network for mammographic mass segmentation which employs a fully convolutional network (FCN) to model a potential function, followed by a CRF to perform structured learning. Because the mass distribution varies greatly with pixel position, the FCN is combined with a position priori. Further, we employ adversarial training to eliminate over-fitting due to the small sizes of mammogram datasets. Multi-scale FCN is employed to improve the segmentation performance. Experimental results on two public datasets, INbreast and DDSM-BCRP, demonstrate that our end-to-end network achieves better performance than state-of-the-art approaches. \footnote{https://github.com/wentaozhu/adversarial-deep-structural-networks.git}
Open AccessAdversarial Deep Structured Nets for Mass Segmentation from Mammograms
Mass segmentation provides effective morphological features which are important for mass diagnosis. In this work, we propose a novel end-to-end network for mammographic mass segmentation which employs a fully convolutional network (FCN) to model a potential function, followed by a CRF to perform structured learning. Because the mass distribution varies greatly with pixel position, the FCN is combined with a position priori. Further, we employ adversarial training to eliminate over-fitting due to the small sizes of mammogram datasets. Multi-scale FCN is employed to improve the segmentation performance. Experimental results on two public datasets, INbreast and DDSM-BCRP, demonstrate that our end-to-end network achieves better performance than state-of-the-art approaches. \footnote{https://github.com/wentaozhu/adversarial-deep-structural-networks.git}
- Open Access
User Experience of the CoSTAR System for Instruction of Collaborative Robots
How can we enable novice users to create effective task plans for collaborative robots? Must there be a tradeoff between generalizability and ease of use? To answer these questions, we conducted a user study with the CoSTAR system, which integrates perception and reasoning into a Behavior Tree-based task plan editor. In our study, we ask novice users to perform simple pick-and-place assembly tasks under varying perception and planning capabilities. Our study shows that users found Behavior Trees to be an effective way of specifying task plans. Furthermore, users were also able to more quickly, effectively, and generally author task plans with the addition of CoSTAR's planning, perception, and reasoning capabilities. Despite these improvements, concepts associated with these capabilities were rated by users as less usable, and our results suggest a direction for further refinement.
Open AccessUser Experience of the CoSTAR System for Instruction of Collaborative Robots
How can we enable novice users to create effective task plans for collaborative robots? Must there be a tradeoff between generalizability and ease of use? To answer these questions, we conducted a user study with the CoSTAR system, which integrates perception and reasoning into a Behavior Tree-based task plan editor. In our study, we ask novice users to perform simple pick-and-place assembly tasks under varying perception and planning capabilities. Our study shows that users found Behavior Trees to be an effective way of specifying task plans. Furthermore, users were also able to more quickly, effectively, and generally author task plans with the addition of CoSTAR's planning, perception, and reasoning capabilities. Despite these improvements, concepts associated with these capabilities were rated by users as less usable, and our results suggest a direction for further refinement.
- Open Access
Do What I Want, Not What I Did: Imitation of Skills by Planning Sequences of Actions
We propose a learning-from-demonstration approach for grounding actions from expert data and an algorithm for using these actions to perform a task in new environments. Our approach is based on an application of sampling-based motion planning to search through the tree of discrete, high-level actions constructed from a symbolic representation of a task. Recursive sampling-based planning is used to explore the space of possible continuous-space instantiations of these actions. We demonstrate the utility of our approach with a magnetic structure assembly task, showing that the robot can intelligently select a sequence of actions in different parts of the workspace and in the presence of obstacles. This approach can better adapt to new environments by selecting the correct high-level actions for the particular environment while taking human preferences into account.
Open AccessDo What I Want, Not What I Did: Imitation of Skills by Planning Sequences of Actions
We propose a learning-from-demonstration approach for grounding actions from expert data and an algorithm for using these actions to perform a task in new environments. Our approach is based on an application of sampling-based motion planning to search through the tree of discrete, high-level actions constructed from a symbolic representation of a task. Recursive sampling-based planning is used to explore the space of possible continuous-space instantiations of these actions. We demonstrate the utility of our approach with a magnetic structure assembly task, showing that the robot can intelligently select a sequence of actions in different parts of the workspace and in the presence of obstacles. This approach can better adapt to new environments by selecting the correct high-level actions for the particular environment while taking human preferences into account.
- 17 April 2015
- Published by Springer Nature in International Journal of Computer Assisted Radiology and Surgery
- Vol. 10 (6) , 981-991
Automated objective surgical skill assessment in the operating room from unstructured tool motion in septoplasty
Previous work on surgical skill assessment using intraoperative tool motion has focused on highly structured surgical tasks such as cholecystectomy and used generic motion metrics such as time and number of movements. Other statistical methods such as hidden Markov models (HMM) and descriptive curve coding (DCC) have been successfully used to assess skill in structured activities on bench-top tasks. Methods to assess skill and provide effective feedback to trainees for unstructured surgical tasks in the operating room, such as tissue dissection in septoplasty, have yet to be developed. We proposed a method that provides a descriptive structure for septoplasty by automatically segmenting it into higher-level meaningful activities called strokes. These activities characterize the surgeon’s tool motion pattern. We constructed a spatial graph from the sequence of strokes in each procedure and used its properties to train a classifier to distinguish between expert and novice surgeons. We compared the results from our method with those from HMM, DCC, and generic metric-based approaches. We showed that our method—with an average accuracy of 91 %—performs better or equal than these state-of-the-art methods, while simultaneously providing surgeons with an intuitive understanding of the procedure. In this study, we developed and evaluated an automated approach to objectively assess surgical skill during unstructured task of tissue dissection in nasal septoplasty.
- 17 April 2015
- Published by Springer Nature in International Journal of Computer Assisted Radiology and Surgery
- Vol. 10 (6) , 981-991
Automated objective surgical skill assessment in the operating room from unstructured tool motion in septoplasty
Previous work on surgical skill assessment using intraoperative tool motion has focused on highly structured surgical tasks such as cholecystectomy and used generic motion metrics such as time and number of movements. Other statistical methods such as hidden Markov models (HMM) and descriptive curve coding (DCC) have been successfully used to assess skill in structured activities on bench-top tasks. Methods to assess skill and provide effective feedback to trainees for unstructured surgical tasks in the operating room, such as tissue dissection in septoplasty, have yet to be developed. We proposed a method that provides a descriptive structure for septoplasty by automatically segmenting it into higher-level meaningful activities called strokes. These activities characterize the surgeon’s tool motion pattern. We constructed a spatial graph from the sequence of strokes in each procedure and used its properties to train a classifier to distinguish between expert and novice surgeons. We compared the results from our method with those from HMM, DCC, and generic metric-based approaches. We showed that our method—with an average accuracy of 91 %—performs better or equal than these state-of-the-art methods, while simultaneously providing surgeons with an intuitive understanding of the procedure. In this study, we developed and evaluated an automated approach to objectively assess surgical skill during unstructured task of tissue dissection in nasal septoplasty.
- Open Access
- 1 January 2014
- Published by Springer Nature
- Vol. 17, 684-691
Needle Guidance Using Handheld Stereo Vision and Projection for Ultrasound-Based Interventions
With real-time instrument tracking and in-situ guidance projection directly integrated in a handheld ultrasound imaging probe, needle-based interventions such as biopsies become much simpler to perform than with conventionally-navigated systems. Stereo imaging with needle detection can be made sufficiently robust and accurate to serve as primary navigation input. We describe the low-cost, easy-to-use approach used in the Clear Guide ONE generic navigation accessory for ultrasound machines, outline different available guidance methods, and provide accuracy results from phantom trials.
- 1 January 2014
- Published by Springer Nature
- Vol. 17, 684-691
Open AccessNeedle Guidance Using Handheld Stereo Vision and Projection for Ultrasound-Based Interventions
With real-time instrument tracking and in-situ guidance projection directly integrated in a handheld ultrasound imaging probe, needle-based interventions such as biopsies become much simpler to perform than with conventionally-navigated systems. Stereo imaging with needle detection can be made sufficiently robust and accurate to serve as primary navigation input. We describe the low-cost, easy-to-use approach used in the Clear Guide ONE generic navigation accessory for ultrasound machines, outline different available guidance methods, and provide accuracy results from phantom trials.