Chuxuan Hu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Chuxuan Hu has a total of 7 co-authors. Here are the top co-authors by number of shared publications:
Aum Wadhwani
University of Illinois Urbana-Champaign
2 shared publications- Austin Peters
University of Chicago
2 shared publications - Daniel Kang
University of Illinois (UIUC) Urbana USA
2 shared publications - Liyun Zhang
Shanghai Jiao Tong University
2 shared publications - Yeji Lim
University of Illinois Urbana-Champaign
2 shared publications - Hanghang Tong
University of Illinois Urbana-Champaign, Urbana, IL, USA
1 shared publication
Latest Publications
- Open Access
REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?
Assessing the reproducibility of social science papers is essential for promoting rigor in research processes, but manual assessment is costly. With recent advances in agentic AI systems (i.e., AI agents), we seek to evaluate their capability to automate this process. However, existing benchmarks for reproducing research papers (1) focus solely on reproducing results using provided code and data without assessing their consistency with the paper, (2) oversimplify real-world scenarios, and (3) lack necessary diversity in data formats and programming languages. To address these issues, we introduce REPRO-Bench, a collection of 112 task instances, each representing a social science paper with a publicly available reproduction report. The agents are tasked with assessing the reproducibility of the paper based on the original paper PDF and the corresponding reproduction package. REPRO-Bench features end-to-end evaluation tasks on the reproducibility of social science papers with complexity comparable to real-world assessments. We evaluate three representative AI agents on REPRO-Bench, with the best-performing agent achieving an accuracy of only 21.4%. Building on our empirical analysis, we develop REPRO-Agent, which improves the highest accuracy achieved by existing agents by 71%. We conclude that more advanced AI agents should be developed to automate real-world reproducibility assessment. REPRO-Bench is publicly available at https://github.com/uiuc-kang-lab/REPRO-Bench.
Open AccessREPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?
Assessing the reproducibility of social science papers is essential for promoting rigor in research processes, but manual assessment is costly. With recent advances in agentic AI systems (i.e., AI agents), we seek to evaluate their capability to automate this process. However, existing benchmarks for reproducing research papers (1) focus solely on reproducing results using provided code and data without assessing their consistency with the paper, (2) oversimplify real-world scenarios, and (3) lack necessary diversity in data formats and programming languages. To address these issues, we introduce REPRO-Bench, a collection of 112 task instances, each representing a social science paper with a publicly available reproduction report. The agents are tasked with assessing the reproducibility of the paper based on the original paper PDF and the corresponding reproduction package. REPRO-Bench features end-to-end evaluation tasks on the reproducibility of social science papers with complexity comparable to real-world assessments. We evaluate three representative AI agents on REPRO-Bench, with the best-performing agent achieving an accuracy of only 21.4%. Building on our empirical analysis, we develop REPRO-Agent, which improves the highest accuracy achieved by existing agents by 71%. We conclude that more advanced AI agents should be developed to automate real-world reproducibility assessment. REPRO-Bench is publicly available at https://github.com/uiuc-kang-lab/REPRO-Bench.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 23616-23626
REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?
Chuxuan Hu, Liyun Zhang, Yeji Lim, Aum Wadhwani, Austin Peters, Daniel Kang. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 23616-23626
Open AccessREPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?
Chuxuan Hu, Liyun Zhang, Yeji Lim, Aum Wadhwani, Austin Peters, Daniel Kang. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
- Open Access
GENIUS: A Novel Solution for Subteam Replacement with Clustering-based Graph Neural Network
Subteam replacement is defined as finding the optimal candidate set of people who can best function as an unavailable subset of members (i.e., subteam) for certain reasons (e.g., conflicts of interests, employee churn), given a team of people embedded in a social network working on the same task. Prior investigations on this problem incorporate graph kernel as the optimal criteria for measuring the similarity between the new optimized team and the original team. However, the increasingly abundant social networks reveal fundamental limitations of existing methods, including (1) the graph kernel-based approaches are powerless to capture the key intrinsic correlations among node features, (2) they generally search over the entire network for every member to be replaced, making it extremely inefficient as the network grows, and (3) the requirement of equal-sized replacement for the unavailable subteam can be inapplicable due to limited hiring budget. In this work, we address the limitations in the state-of-the-art for subteam replacement by (1) proposing GENIUS, a novel clustering-based graph neural network (GNN) framework that can capture team network knowledge for flexible subteam replacement, and (2) equipping the proposed GENIUS with self-supervised positive team contrasting training scheme to improve the team-level representation learning and unsupervised node clusters to prune candidates for fast computation. Through extensive empirical evaluations, we demonstrate the efficacy of the proposed method (1) effectiveness: being able to select better candidate members that significantly increase the similarity between the optimized and original teams, and (2) efficiency: achieving more than 600 times speed-up in average running time.
Open AccessGENIUS: A Novel Solution for Subteam Replacement with Clustering-based Graph Neural Network
Subteam replacement is defined as finding the optimal candidate set of people who can best function as an unavailable subset of members (i.e., subteam) for certain reasons (e.g., conflicts of interests, employee churn), given a team of people embedded in a social network working on the same task. Prior investigations on this problem incorporate graph kernel as the optimal criteria for measuring the similarity between the new optimized team and the original team. However, the increasingly abundant social networks reveal fundamental limitations of existing methods, including (1) the graph kernel-based approaches are powerless to capture the key intrinsic correlations among node features, (2) they generally search over the entire network for every member to be replaced, making it extremely inefficient as the network grows, and (3) the requirement of equal-sized replacement for the unavailable subteam can be inapplicable due to limited hiring budget. In this work, we address the limitations in the state-of-the-art for subteam replacement by (1) proposing GENIUS, a novel clustering-based graph neural network (GNN) framework that can capture team network knowledge for flexible subteam replacement, and (2) equipping the proposed GENIUS with self-supervised positive team contrasting training scheme to improve the team-level representation learning and unsupervised node clusters to prune candidates for fast computation. Through extensive empirical evaluations, we demonstrate the efficacy of the proposed method (1) effectiveness: being able to select better candidate members that significantly increase the similarity between the optimized and original teams, and (2) efficiency: achieving more than 600 times speed-up in average running time.