HWHongtao Wu

Hongtao Wu

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Hongtao Wu has a total of 36 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Top Cited Papers

    GR-MG: Leveraging Partially-Annotated Data via Multi-Modal Goal-Conditioned Policy

    by Peiyan Li, Hongtao Wu, Yan Huang, Chilam Cheang, Liang Wang, Tao Kong

    The robotics community has consistently aimed to achieve generalizable robot manipulation with flexible natural language instructions. One primary challenge is that obtaining robot trajectories fully annotated with both actions and texts is time-consuming and labor-intensive. However, partially-annotated data, such as human activity videos without action labels and robot trajectories without text labels, are much easier to collect. Can we leverage these data to enhance the generalization capabilities of robots? In this letter, we propose GR-MG, a novel method which supports conditioning on a text instruction and a goal image. During training, GR-MG samples goal images from trajectories and conditions on both the text and the goal image or solely on the image when text is not available. During inference, where only the text is provided, GR-MG generates the goal image via a diffusion-based image-editing model and conditions on both the text and the generated image. This approach enables GR-MG to leverage large amounts of partially-annotated data while still using languages to flexibly specify tasks. To generate accurate goal images, we propose a novel progress-guided goal image generation model which injects task progress information into the generation process. In simulation experiments, GR-MG improves the average number of tasks completed in a row of 5 from 3.35 to 4.04. In real-robot experiments, GR-MG is able to perform 58 different tasks and improves the success rate from 68.7% to 78.1% and 44.4% to 60.6% in simple and generalization settings, respectively. It also outperforms comparing baseline methods in few-shot learning of novel skills.


  • Open Access

    Prepare the Chair for the Bear! Robot Imagination of Sitting Affordance to Reorient Previously Unseen Chairs

    by Xin Meng, Hongtao Wu, Sipu Ruan, Gregory S. Chirikjian

    In this letter, a paradigm for the classification and manipulation of novel objects is established and demonstrated with the example of chairs. Our approach leverages the robot's understanding of object stability, perceptibility, and affordance to prepare previously unseen and randomly oriented chairs on which a teddy bear is to be seated. The teddy bear is a proxy for an elderly person, hospital patient, or child. By autonomously reconstructing a complete model of the object and inserting it into a physical simulator ( i.e. , the robot's “imagination”), the robot assesses whether or not the object is a chair and, if it is, determines how to reorient it properly to be used. Experimental results show that our method achieves a high success rate on the real robot task of chair preparation. Also, it outperforms several baseline methods on the task of upright pose prediction for chairs. The same methodology can be easily transferred to a wide variety of application scenarios, and illustrates a broader paradigm in affordance-based reasoning.


  • A Method for Detecting Road Spills Under the Constraint of Short Term-Long Term Dual Background Model

    by Ying Meng, Hongtao Wu, Bingqing Niu, Junyi Ren, Mingshu Shen

    With the development of urban traffic and the occurrence of a large number of traffic accidents, the use of video-based traffic event detection instead of human real-time monitoring of traffic scenes, automatic detection and identification of road obstacles such as spills that may cause traffic accidents has become a hot spot in the research of intelligent transportation systems. This paper addresses the problem of low accuracy of existing video surveillance spills detection, proposes a short term-long term dual background model constrained road spills detection method and applies it to the detection of road spills traffic events, combines the classification recognition results of classifier to eliminate misjudgment and realize the event detection of road spills, and also realizes the 3D information recovery of spills and extracts 3D height information to complete the spills target recognition. It has good application value for constructing a perfect highway event detection, management and diversion system. At the same time, the research of this paper also lays a good foundation for the subsequent work of traffic event detection, computer vision reconstruction, etc. It has important guiding significance and reference value.


  • Open Access

    Prepare the Chair for the Bear! Robot Imagination of Sitting Affordance to Reorient Previously Unseen Chairs

    by Xin Meng, Hongtao Wu, Sipu Ruan, Gregory S. Chirikjian

    In this letter, a paradigm for the classification and manipulation of previously unseen objects is established and demonstrated through a real example of chairs. We present a novel robot manipulation method, guided by the understanding of object stability, perceptibility, and affordance, which allows the robot to prepare previously unseen and randomly oriented chairs for a teddy bear to sit on. Specifically, the robot encounters an unknown object and first reconstructs a complete 3D model from perceptual data via active and autonomous manipulation. By inserting this model into a physical simulator (i.e., the robot's "imagination"), the robot assesses whether the object is a chair and determines how to reorient it properly to be used, i.e., how to reorient it to an upright and accessible pose. If the object is classified as a chair, the robot reorients the object to this pose and seats the teddy bear onto the chair. The teddy bear is a proxy for an elderly person, hospital patient, or child. Experiment results show that our method achieves a high success rate on the real robot task of chair preparation. Also, it outperforms several baseline methods on the task of upright pose prediction for chairs.

  • Open Access

    Efficient Path Planning in Narrow Passages for Robots With Ellipsoidal Components

    by Sipu Ruan, Karen L. Poblete, Hongtao Wu, Qianli Ma, Gregory S. Chirikjian

    Path planning has long been one of the major research areas in robotics, with probabilistic roadmap (PRM) and rapidly-exploring random trees (RRT) being two of the most effective classes of planners. Though generally very efficient, these sampling-based planners can become computationally expensive in the important case of “narrow passages.” This article develops a path planning paradigm specifically formulated for narrow passage problems. The core is based on planning for rigid-body robots encapsulated by unions of ellipsoids. Each environmental feature is represented geometrically using a strictly convex body with a $\mathcal {C}^{1}$ boundary (e.g., superquadric). The main benefit of doing this is that configuration-space obstacles can be parameterized explicitly in closed form, thereby allowing prior knowledge to be used to avoid sampling infeasible configurations. Then, by characterizing a tight volume bound for multiple ellipsoids, robot transitions involving rotations are guaranteed to be collision free without needing to perform traditional collision detection. Furthermore, by combining with a stochastic sampling strategy, the proposed planning framework can be extended to solving higher dimensional problems, in which the robot has a moving base and articulated appendages. Benchmark results show that the proposed framework often outperforms the sampling-based planners in terms of computational time and success rate in finding a path through narrow corridors for both single-body robots and those with higher dimensional configuration spaces. Physical experiments using the proposed framework are further demonstrated on a humanoid robot that walks in several cluttered environments with narrow passages.


  • Open Access

    LSG-CPD: Coherent Point Drift with Local Surface Geometry for Point Cloud Registration

    by Weixiao Liu, Hongtao Wu, Gregory S. Chirikjian

    Probabilistic point cloud registration methods are becoming more popular because of their robustness. However, unlike point-to-plane variants of iterative closest point (ICP) which incorporate local surface geometric information such as surface normals, most probabilistic methods (e.g., coherent point drift (CPD)) ignore such information and build Gaussian mixture models (GMMs) with isotropic Gaussian covariances. This results in sphere-like GMM components which only penalize the point-to-point distance between the two point clouds. In this paper, we propose a novel method called CPD with Local Surface Geometry (LSG-CPD) for rigid point cloud registration. Our method adaptively adds different levels of point-to-plane penalization on top of the point-to-point penalization based on the flatness of the local surface. This results in GMM components with anisotropic covariances. We formulate point cloud registration as a maximum likelihood estimation (MLE) problem and solve it with the Expectation-Maximization (EM) algorithm. In the E step, we demonstrate that the computation can be recast into simple matrix manipulations and efficiently computed on a GPU. In the M step, we perform an unconstrained optimization on a matrix Lie group to efficiently update the rigid transformation of the registration. The proposed method outperforms state-of-the-art algorithms in terms of accuracy and robustness on various datasets captured with range scanners, RGBD cameras, and LiDARs. Also, it is significantly faster than modern implementations of CPD. The source code is available at https://github.com/ChirikjianLab/LSG-CPD.git.


  • Can I Pour Into It? Robot Imagining Open Containability Affordance of Previously Unseen Objects via Physical Simulations

    by Hongtao Wu, Gregory S. Chirikjian

    Open containers, i.e., containers without covers, are an important and ubiquitous class of objects in human life. In this letter, we propose a novel method for robots to "imagine" the open containability affordance of a previously unseen object via physical simulations. We implement our imagination method on a UR5 manipulator. The robot autonomously scans the object with an RGB-D camera. The scanned 3D model is used for open containability imagination which quantifies the open containability affordance by physically simulating dropping particles onto the object and counting how many particles are retained in it. This quantification is used for open-container vs. non-open-container binary classification (hereafter referred to as open container classification). If the object is classified as an open container, the robot further imagines pouring into the object, again using physical simulations, to obtain the pouring position and orientation for real robot autonomous pouring. We evaluate our method on open container classification and autonomous pouring of granular material on a dataset containing 130 previously unseen objects with 57 object categories. Although our proposed method uses only 11 objects for simulation calibration (training), its open container classification aligns well with human judgements. In addition, our method endows the robot with the capability to autonomously pour into the 55 containers in the dataset with a very high success rate. We also compare to a deep learning method. Results show that our method achieves the same performance as the deep learning method on open container classification and outperforms it on autonomous pouring. Moreover, our method is fully explainable.


  • Open Access

    Steroid receptor RNA activator inhibits the migration, invasion and stemness characteristics of renal cell carcinoma cells

    by Chanjuan Zhang, Neng Zhu, Chan Liu, Hongtao Wu, Yantao Yin, Yaning Shi, Duanfang Liao, Li Qin

    Renal cell carcinoma (RCC) has a high mortality rate among urological malignancies, and its underlying mechanisms remain unclear. Steroid receptor RNA coactivator (SRA) belongs to the long non‑coding RNAs (lncRNAs) and has been demonstrated to be closely related to various types of cancer. In the present study, the decreased expression level of SRA was first confirmed in RCC tissues and cell lines by RT‑qPCR. Using knockdown or overexpression systems, it was then found that SRA inhibited the proliferation of RCC cell lines and promoted their apoptosis. In addition, SRA suppressed the migration and invasion, and altered EMT‑related markers in RCC cells. More importantly, it was demonstrated that SRA reduced percentage of CD44+/CD24‑ cells and the sphere‑forming efficiency. SRA also attenuated the expression levels of CD44, SOX‑2, ABCG2 and OCT‑4, which are all associated with cancer cell stemness characteristics. Although SRA increased the phosphorylation of extracellular‑regulated protein kinase (ERK), the ERK1/2 pathway could not further interfere with the alteration of EMT‑related markers mediated by SRA. Notably, the ERK inhibitor, PD98059, abolished ERK1/2 phosphorylation, whereas it did not exert any marked effects on cell proliferation and EMT‑related markers mediated by SRA. Taken together, the findings of the present study indicate that SRA is an important molecule that inhibits the migration, invasion and stem cell characteristics of RCC cells; the ERK signaling pathway may not be involved in this process.


  • “Good Robot!”: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer

    by Andrew Hundt, Benjamin Killeen, Nicholas Greene, Hongtao Wu, Heeyeon Kwon, Chris Paxton, Gregory D. Hager

    Current Reinforcement Learning (RL) algorithms struggle with long-horizon tasks where time can be wasted exploring dead ends and task progress may be easily reversed. We develop the SPOT framework, which explores within action safety zones, learns about unsafe regions without exploring them, and prioritizes experiences that reverse earlier progress to learn with remarkable efficiency. The SPOT framework successfully completes simulated trials of a variety of tasks, improving a baseline trial success rate from 13% to 100% when stacking 4 cubes, from 13% to 99% when creating rows of 4 cubes, and from 84% to 95% when clearing toys arranged in adversarial patterns. Efficiency with respect to actions per trial typically improves by 30% or more, while training takes just 1-20k actions, depending on the task. Furthermore, we demonstrate direct sim to real transfer. We are able to create real stacks in 100% of trials with 61% efficiency and real rows in 100% of trials with 59% efficiency by directly loading the simulation-trained model on the real robot with no additional real-world fine-tuning. To our knowledge, this is the first instance of reinforcement learning with successful sim to real transfer applied to long term multi-step tasks such as block-stacking and row-making with consideration of progress reversal. Code is available at https://github.com/jhu-lcsr/good_robot.


  • Multi-Mosquito Object Detection and 2D Pose Estimation for Automation of PfSPZ Malaria Vaccine Production

    by Hongtao Wu, Jiteng Mu, Ting Da, Mengdi Xu, Russell H. Taylor, Iulian Iordachita, Gregory S. Chirikjian

    Multi-mosquito object detection and 2D pose estimation are essential steps towards fully automated extracting PfSPZ-infected mosquito salivary glands for manufacture of PfSPZ Vaccine, which has been shown to protect against malaria in multiple clinical trials in the US, Europe, and Africa. This paper presents a deep learning approach to perform cluster condition classification and bounding box detection of multiple mosquitoes in an image. It also estimates the 2D pose of each non-clustered mosquito by body part detection. This approach is based on two popular convolutional neural network (CNN) architectures, Mask R-CNN and DeeperCut. In addition, we propose a cascaded image processing approach to achieve the multi-mosquito detection, cluster condition classification, and body parts detection in a multi-step manner. We compare the two approaches in terms of their functionality, robustness, accuracy, and speed. We hope our effective approaches would push forward the automation of PfSPZ Vaccine production to facilitate the prevention and elimination of this disease worldwide.