HFHao Fang

Hao Fang

*This is an algorithmically generated author record based on publicly available information.

Hao Fang has a total of 12 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error

    by Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, Yu Su

    Tools are essential for large language models (LLMs) to acquire up-to-date information and take consequential actions in external environments. Existing work on tool-augmented LLMs primarily focuses on the broad coverage of tools and the flexibility of adding new tools. However, a critical aspect that has surprisingly been understudied is simply how accurately an LLM uses tools for which it has been trained. We find that existing LLMs, including GPT-4 and open-source LLMs specifically fine-tuned for tool use, only reach a correctness rate in the range of 30% to 60%, far from reliable use in practice. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE), that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. Specifically, STE leverages an LLM's 'imagination' to simulate plausible scenarios for using a tool, after which the LLM interacts with the tool to learn from its execution feedback. Both short-term and long-term memory are employed to improve the depth and breadth of the exploration, respectively. Comprehensive experiments on ToolBench show that STE substantially improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform GPT-4. We also show effective continual learning of tools via a simple experience replay strategy.


  • Top Cited Papers
    Open Access

    LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error

    by Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, Yu Su

    Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, Yu Su. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.


  • Open Access

    Few-Shot Adaptation for Parsing Contextual Utterances with LLMs

    by Kevin Lin, Patrick Xia, Hao Fang

    We evaluate the ability of semantic parsers based on large language models (LLMs) to handle contextual utterances. In real-world settings, there typically exists only a limited number of annotated contextual utterances due to annotation cost, resulting in an imbalance compared to non-contextual utterances. Therefore, parsers must adapt to contextual utterances with a few training examples. We examine four major paradigms for doing so in conversational semantic parsing i.e., Parse-with-Utterance-History, Parse-with-Reference-Program, Parse-then-Resolve, and Rewrite-then-Parse. To facilitate such cross-paradigm comparisons, we construct SMCalFlow-EventQueries, a subset of contextual examples from SMCalFlow with additional annotations. Experiments with in-context learning and fine-tuning suggest that Rewrite-then-Parse is the most promising paradigm when holistically considering parsing accuracy, annotation cost, and error types.

  • Open Access

    Few-Shot Adaptation for Parsing Contextual Utterances with LLMs

    by Kevin Lin, Patrick Xia, Hao Fang

    Kevin Lin, Patrick Xia, Hao Fang. Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings). 2023.


  • Open Access

    MR-CapsNet: A Deep Learning Algorithm for Image-Based Head Pose Estimation on CapsNet

    by Hao Fang, Jun-Qing Liu, Kai Xie, Peng Wu, Xin-Yu Zhang, Chang Wen, Jian-Biao He

    Head pose estimation based on a single image is a challenging endeavor because of the complex background conditions and characteristics of the human face. In this report, we propose a Multi stage Regression-Capsule Network (MR-CapsNet) to predict head posture based on a single image input. In the study, we used the residual attention block and squeeze-and-excitation block to extract features in three levels. CapsNet overcomes the shortcomings of the traditional convolutional neural network and implements module aggregation to describe the spatial relationship of features after aggregation, in addition to realizing a compact and robust model using a multi-stage regression scheme. We tested our method on the AFLW2000 and BIWI datasets obtaining mean absolute errors of 4.26% and 3.95%, respectively. In addition, we discuss the accuracy of our method in the case of eye or mouth occlusion. The results of comprehensive experiments reveal that our method can accurately predict head posture.