MZMuhan Zhang

Muhan Zhang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Muhan Zhang has a total of 114 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Position: how can graphs help large language models?

    by Xiyuan Wang, Yi Hu, Yanbo Wang, Chuan Shi, Muhan Zhang

  • Open Access

    In-Parameter Learning: Why Lifelong AI Systems Need More Than Longer Context

    by Yiding Wang, Haotong Yang, Pingzhi Tang, Shijia Kang, Cai Zhou, Chenglin Zhu, Fan Jiang, Fanxu Meng, Juntong Wang, Libin Chen, et al.

    AI agents are increasingly expected to interact with changing users, tools, and environments over extended periods of time. However, current systems remain largely static: their parameters are fixed during deployment, while new experiences, domain knowledge, corrections, and behavioral adaptations are handled primarily through external memory management and in-context learning. In this position paper, we focus on lifelong AI: AI systems that can continually learn from their long-term experience throughout an extended deployment lifetime. This position paper argues that existing in-context mechanisms are insufficient for lifelong AI, and advocate In-Parameter Learning (IPL) as a necessary complement to In-Context Learning (ICL), referring to mechanisms that allow a deployed model to continually and safely update its parameters during use. More broadly, we aim to renew the community’s attention to in-parameter learning and to recalibrate the overly optimistic expectations placed on in-context learning alone. Our central claim is that future lifelong AI systems should be built around a hybrid paradigm: in-context learning for immediate, provisional, and reversible information, and in-parameter learning for durable, cumulative, and generalizable growth.

  • Open Access

    Are Graphs Useful for LLMs? A Comprehensive Survey of Graph-Enhanced Large Language Models

    by Xinyan Zhu, Cheng Yang, Qiuyu Wang, Zeyuan Guo, Zedi Liu, Yiding Wang, Jiawei Liu, Chunchen Wang, Muhan Zhang, Chuan Shi

    Large language models (LLMs) have demonstrated strong capabilities across diverse tasks such as language understanding, reasoning, planning, and code generation. However, their sequence-based architectures limit their ability to capture complex relational structures, long-range dependencies, and multi-hop interactions. Graphs, which explicitly model entities and relationships, provide a natural complement to LLMs by enabling structured representation, multi-hop reasoning, and improved knowledge grounding. This synergy has led to a growing body of research on graph-enhanced LLMs, which we refer to as Graph4LLM. In this survey, we present a systematic, pipeline-oriented review of Graph4LLM methods, categorizing them into three stages of the LLM pipeline: (1) the input phase, where graphs structure prompts and incorporate external knowledge; (2) the model phase, where graphs guide word-level representations and agent-level coordination; and (3) the output phase, where graphs support structured reasoning, planning, and verification. For each phase, we provide a detailed review of the key methods and techniques. We further present a broad range of application scenarios, organizing them into general and domain-specific applications, and highlight how Graph4LLM methods demonstrate strong potential across diverse tasks and real-world settings. Finally, we outline the challenges and future research directions for developing more efficient and interpretable solutions. Resources for Graph4LLM are available at https://github.com/BUPT-GAMMA/Awesome-Graph4LLM.

  • Open Access

    GREPO: A Benchmark for Graph Neural Networks on Repository-Level Bug Localization

    by Juntong Wang, Libin Chen, Xiyuan Wang, Shijia Kang, Haotong Yang, Da Zheng, Muhan Zhang

    Repository-level bug localization-the task of identifying where code must be modified to fix a bug-is a critical software engineering challenge. Standard Large Language Modles (LLMs) are often unsuitable for this task due to context window limitations that prevent them from processing entire code repositories. As a result, various retrieval methods are commonly used, including keyword matching, text similarity, and simple graph-based heuristics such as Breadth-First Search. Graph Neural Networks (GNNs) offer a promising alternative due to their ability to model complex, repository-wide dependencies; however, their application has been hindered by the lack of a dedicated benchmark. To address this gap, we introduce GREPO, the first GNN benchmark for repository-scale bug localization tasks. GREPO comprises 86 Python repositories and 47294 bug-fixing tasks, providing graph-based data structures ready for direct GNN processing. Our evaluation of various GNN architectures shows outstanding performance compared to established information retrieval baselines. This work highlights the potential of GNNs for bug localization and established GREPO as a foundation resource for future research, The code is available at https://github.com/qingpingmo/GREPO.

  • Open Access

    SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass

    by Yewei Liu, Xiyuan Wang, Yansheng Mao, Yoav Gelbery, Haggai Maron, Muhan Zhang

    We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLM). By reusing the frozen LLM's own parameters in an in-context hypernetwork design and introducing architectural innovations, SHINE overcomes key limitations of prior hypernetworks and achieves strong expressive power with a relatively small number of parameters. We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters from diverse meaningful contexts in a single forward pass. It updates LLM parameters without any fine-tuning, and immediately enables complex question answering tasks related to the context without directly accessing the context, effectively transforming in-context knowledge to in-parameter knowledge in one pass. Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling. Our code is available at https://github.com/Yewei-Liu/SHINE


  • Open Access

    LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers

    by Yike Sun, Haotong Yang, Zhouchen Lin, Muhan Zhang

    Tokenization is fundamental to how language models represent and process text, yet the behavior of widely used BPE tokenizers has received far less study than model architectures and training. In this paper, we investigate intermediate merge residues in BPE vocabularies: tokens that are frequent during merge learning so that retained in the final vocabulary, but are mostly further merged and rarely emitted when tokenizing the corpus during tokenizer usage. Such low-frequency tokens not only waste vocabulary capacity but also increase vulnerability to adversarial or atypical inputs. We present a systematic empirical characterization of this phenomenon across commonly used tokenizers and introduce LiteToken, a simple method for removing residue tokens. Because the affected tokens are rarely used, pretrained models can often accommodate the modified tokenizer without additional fine-tuning. Experiments show that LiteToken reduces token fragmentation, reduces parameters, and improves robustness to noisy or misspelled inputs, while preserving overall performance.

  • Open Access

    Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation

    by Pingzhi Tang, Ruijie Zhou, Fanxu Meng, Wenjie Pei, Muhan Zhang

    Current quantization methods for LLMs predominantly rely on block-wise structures to maintain efficiency, often at the cost of representational flexibility. In this work, we demonstrate that element-wise quantization can be made as efficient as block-wise scaling while providing strictly superior expressive power by modeling the scaling manifold as continuous low-rank matrices ($S = BA$). We propose Low-Rank Decomposed Scaling (LoRDS), a unified framework that rethinks quantization granularity through this low-rank decomposition. By "breaking the blocks" of spatial constraints, LoRDS establishes a seamless efficiency lifecycle: it provides high-fidelity PTQ initialization refined via iterative optimization, enables joint QAT of weights and scaling factors, and facilitates high-rank multiplicative PEFT adaptation. Unlike additive PEFT approaches such as QLoRA, LoRDS enables high-rank weight updates within a low-rank budget while incurring no additional inference overhead. Supported by highly optimized Triton kernels, LoRDS consistently outperforms state-of-the-art baselines across various model families in both quantization and downstream fine-tuning tasks. Notably, on Llama3-8B, our method achieves up to a 27.0% accuracy improvement at 3 bits over NormalFloat quantization and delivers a 1.5x inference speedup on NVIDIA RTX 4090 while enhancing PEFT performance by 9.6% on downstream tasks over 4bit QLoRA, offering a robust and integrated solution for unified compression and adaptation of LLMs.

  • Open Access

    Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation

    by Pingzhi Tang, Yiding Wang, Muhan Zhang

    Large Language Models (LLMs) face the "knowledge cutoff" challenge, where their frozen parametric memory prevents direct internalization of new information. While Supervised Fine-Tuning (SFT) is commonly used to update model knowledge, it often updates factual content without reliably improving the model's ability to use the newly incorporated information for question answering or decision-making. Reinforcement Learning (RL) is essential for acquiring reasoning skills; however, its high computational cost makes it impractical for efficient online adaptation. We empirically observe that the parameter updates induced by SFT and RL are nearly orthogonal. Based on this observation, we propose Parametric Skill Transfer (PaST), a framework that supports modular skill transfer for efficient and effective knowledge adaptation. By extracting a domain-agnostic Skill Vector from a source domain, we can linearly inject knowledge manipulation skills into a target model after it has undergone lightweight SFT on new data. Experiments on knowledge-incorporation QA (SQuAD, LooGLE) and agentic tool-use benchmarks (ToolBench) demonstrate the effectiveness of our method. On SQuAD, PaST outperforms the state-of-the-art self-editing SFT baseline by up to 9.9 points. PaST further scales to long-context QA on LooGLE with an 8.0-point absolute accuracy gain, and improves zero-shot ToolBench success rates by +10.3 points on average with consistent gains across tool categories, indicating strong scalability and cross-domain transferability of the Skill Vector.

  • Open Access

    SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding

    by Shuyang Hou, Yi Hu, Muhan Zhang

    Recent advancements in large language models (LLMs) have significantly enhanced their reasoning capabilities. However, they continue to struggle with basic character-level tasks, such as counting letters in words, a problem rooted in their tokenization process. While existing benchmarks have highlighted this weakness through basic character operations, such failures are often dismissed due to lacking practical relevance. Yet, many real-world applications, such as navigating text-based maps or interpreting structured tables, rely heavily on precise sub-token understanding. In this regard, we introduce SubTokenTest, a comprehensive benchmark that assesses sub-token understanding through practical, utility-driven tasks. Our benchmark includes ten tasks across four domains and isolates tokenization-related failures by decoupling performance from complex reasoning. We provide a comprehensive evaluation of nine advanced LLMs. Additionally, we investigate the impact of test-time scaling on sub-token reasoning and explore how character-level information is encoded within the hidden states.

  • Open Access

    LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges?

    by Ziyuan He, Yuxuan Wang, Jiaqi Li, Kexin Liang, Muhan Zhang

    Large language models (LLMs) are equipped with increasingly extended context windows recently, yet their long context understanding capabilities over long dependency tasks remain fundamentally limited and underexplored. This gap is especially significant in many real-world long-context applications that were rarely benchmarked. In this paper, we introduce LooGLE v2, a novel benchmark designed to evaluate LLMs' long context ability in real-world applications and scenarios. Our benchmark consists of automatically collected real-world long texts, ranging from 16k to 2M tokens, encompassing domains in law, finance, game and code. Accordingly, we delicately design 10 types of domain-specific long-dependency tasks and generate 1,934 QA instances with various diversity and complexity in a scalable data curation pipeline for further practical needs. We conduct a comprehensive assessment of 6 locally deployed and 4 API-based LLMs. The evaluation results show that even the best-performing model achieves only a 59.2% overall score on our benchmark. Despite the extensive context windows, popular LLMs are only capable of understanding a much shorter length of context than they claim to be, revealing significant limitations in their ability to handle real-world tasks with long dependencies and highlighting substantial room for model improvement in practical long-context understanding.