MCMuhao Chen

Muhao Chen

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Muhao Chen has a total of 89 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluation

    by Qin Liu, Jacob Dineen, Yuxi Huang, Sheng Zhang, Hoifung Poon, Ben Zhou, Muhao Chen

    Benchmarks are central to measuring the capabilities of large language models and guiding model development, yet widespread data leakage from pretraining corpora undermines their validity. Models can match memorized content rather than demonstrate true generalization, which inflates scores, distorts cross-model comparisons, and misrepresents progress. We introduce ArenaBencher, a model-agnostic framework for automatic benchmark evolution that updates test cases while preserving comparability. Given an existing benchmark and a diverse pool of models to be evaluated, ArenaBencher infers the core ability of each test case, generates candidate question-answer pairs that preserve the original objective, verifies correctness and intent with an LLM as a judge, and aggregates feedback from multiple models to select candidates that expose shared weaknesses. The process runs iteratively with in-context demonstrations that steer generation toward more challenging and diagnostic cases. We apply ArenaBencher to math problem solving, commonsense reasoning, and safety domains and show that it produces verified, diverse, and fair updates that uncover new failure modes, increase difficulty while preserving test objective alignment, and improve model separability. The framework provides a scalable path to continuously evolve benchmarks in step with the rapid progress of foundation models.

  • Open Access

    Is Extending Modality The Right Path Towards Omni-Modality?

    by Tinghui Zhu, Kai Zhang, Muhao Chen, Yu Su

    Omni-modal language models (OLMs) aim to integrate and reason over diverse input modalities--such as text, images, video, and audio--while maintaining strong language capabilities. Despite recent advancements, existing models, especially open-source ones, remain far from true omni-modality, struggling to generalize beyond the specific modality pairs they are trained on or to achieve strong performance when processing multi-modal inputs. We study the effect of extending modality, the dominant technique for training multimodal models, where an off-the-shelf language model is fine-tuned on target-domain and language data. Specifically, we investigate three key questions: (1) Does modality extension compromise core language abilities? (2) Can model merging effectively integrate independently fine-tuned modality-specific models to achieve omni-modality? (3) Does omni-modality extension lead to better knowledge sharing and generalization compared to sequential extension? Through extensive experiments, we analyze these trade-offs and provide insights into the feasibility of achieving true omni-modality using current approaches.

  • Open Access

    MapQA: Open-domain Geospatial Question Answering on Map Data

    by Zekun Li, Malcolm Grossman, Eric, Ehsan Qasemi, Mihir Kulkarni, Muhao Chen, Yao-Yi Chiang

    Geospatial question answering (QA) is a fundamental task in navigation and point of interest (POI) searches. While existing geospatial QA datasets exist, they are limited in both scale and diversity, often relying solely on textual descriptions of geo-entities without considering their geometries. A major challenge in scaling geospatial QA datasets for reasoning lies in the complexity of geospatial relationships, which require integrating spatial structures, topological dependencies, and multi-hop reasoning capabilities that most text-based QA datasets lack. To address these limitations, we introduce MapQA, a novel dataset that not only provides question-answer pairs but also includes the geometries of geo-entities referenced in the questions. MapQA is constructed using SQL query templates to extract question-answer pairs from OpenStreetMap (OSM) for two study regions: Southern California and Illinois. It consists of 3,154 QA pairs spanning nine question types that require geospatial reasoning, such as neighborhood inference and geo-entity type identification. Compared to existing datasets, MapQA expands both the number and diversity of geospatial question types. We explore two approaches to tackle this challenge: (1) a retrieval-based language model that ranks candidate geo-entities by embedding similarity, and (2) a large language model (LLM) that generates SQL queries from natural language questions and geo-entity attributes, which are then executed against an OSM database. Our findings indicate that retrieval-based methods effectively capture concepts like closeness and direction but struggle with questions that require explicit computations (e.g., distance calculations). LLMs (e.g., GPT and Gemini) excel at generating SQL queries for one-hop reasoning but face challenges with multi-hop reasoning, highlighting a key bottleneck in advancing geospatial QA systems.


  • Open Access

    BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge

    by Terry Tong, Fei Wang, Zhe Zhao, Muhao Chen

    This paper proposes a novel backdoor threat attacking the LLM-as-a-Judge evaluation regime, where the adversary controls both the candidate and evaluator model. The backdoored evaluator victimizes benign users by unfairly assigning inflated scores to adversary. A trivial single token backdoor poisoning 1% of the evaluator training data triples the adversary's score with respect to their legitimate score. We systematically categorize levels of data access corresponding to three real-world settings, (1) web poisoning, (2) malicious annotator, and (3) weight poisoning. These regimes reflect a weak to strong escalation of data access that highly correlates with attack severity. Under the weakest assumptions - web poisoning (1), the adversary still induces a 20% score inflation. Likewise, in the (3) weight poisoning regime, the stronger assumptions enable the adversary to inflate their scores from 1.5/5 to 4.9/5. The backdoor threat generalizes across different evaluator architectures, trigger designs, evaluation tasks, and poisoning rates. By poisoning 10% of the evaluator training data, we control toxicity judges (Guardrails) to misclassify toxic prompts as non-toxic 89% of the time, and document reranker judges in RAG to rank the poisoned document first 97% of the time. LLM-as-a-Judge is uniquely positioned at the intersection of ethics and technology, where social implications of mislead model selection and evaluation constrain the available defensive tools. Amidst these challenges, model merging emerges as a principled tool to offset the backdoor, reducing ASR to near 0% whilst maintaining SOTA performance. Model merging's low computational cost and convenient integration into the current LLM Judge training pipeline position it as a promising avenue for backdoor mitigation in the LLM-as-a-Judge setting.

  • Open Access

    VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap

    by Qin Liu, Fei Wang, Chaowei Xiao, Muhao Chen

    The emergence of vision language models (VLMs) comes with increased safety concerns, as the incorporation of multiple modalities heightens vulnerability to attacks. Although VLMs can be built upon LLMs that have textual safety alignment, it is easily undermined when the vision modality is integrated. We attribute this safety challenge to the modality gap, a separation of image and text in the shared representation space, which blurs the distinction between harmful and harmless queries that is evident in LLMs but weakened in VLMs. To avoid safety decay and fulfill the safety alignment gap, we propose VLM-Guard, an inference-time intervention strategy that leverages the LLM component of a VLM as supervision for the safety alignment of the VLM. VLM-Guard projects the representations of VLM into the subspace that is orthogonal to the safety steering direction that is extracted from the safety-aligned LLM. Experimental results on three malicious instruction settings show the effectiveness of VLM-Guard in safeguarding VLM and fulfilling the safety alignment gap between VLM and its LLM component.


  • Open Access

    Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical KV Equalization

    by Shuyang Hao, Yiwei Wang, Bryan Hooi, Jun Liu, Muhao Chen, Zi Huang, Yujun Cai

    Shuyang Hao, Yiwei Wang, Bryan Hooi, Jun Liu, Muhao Chen, Zi Huang, Yujun Cai. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.


  • Open Access

    Code Execution as Grounded Supervision for LLM Reasoning

    by Dongwon Jung, Wenxuan Zhou, Muhao Chen

    Dongwon Jung, Wenxuan Zhou, Muhao Chen. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.


  • Open Access

    Familiarity-Aware Evidence Compression for Retrieval-Augmented Generation

    by Dongwon Jung, Qin Liu, Tenghao Huang, Ben Zhou, Muhao Chen

    Dongwon Jung, Qin Liu, Tenghao Huang, Ben Zhou, Muhao Chen. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.


  • Open Access

    Active Layer-Contrastive Decoding Reduces Hallucination in Large Language Model Generation

    by Hongxiang Zhang, Hao Chen, Muhao Chen, Tianyi Zhang

    Hongxiang Zhang, Hao Chen, Muhao Chen, Tianyi Zhang. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.


  • Top Cited Papers
    Open Access

    ThinkGuard: Deliberative Slow Thinking Leads to Cautious Guardrails

    by Xiaofei Wen, Wenxuan Zhou, Wenjie Jacky Mo, Muhao Chen

    Xiaofei Wen, Wenxuan Zhou, Wenjie Jacky Mo, Muhao Chen. Findings of the Association for Computational Linguistics: ACL 2025. 2025.