Mingzhe Du
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Mingzhe Du has a total of 35 co-authors. Here are the top co-authors by number of shared publications:
- See-Kiong Ng
National University of Singapore, Singapore, Singapore
10 shared publications - Dong Huang
National University of Singapore, Singapore, Singapore
3 shared publications - Jie M. Zhang
King’s College London, London, United Kingdom of Great Britain and Northern Ireland
3 shared publications - Sujatha Das Gollapalli
Institute of Data Science, National University of Singapore, Singapore, Singapore
3 shared publications - Anh Tuan Luu
Nanyang technological university, Singapore, Singapore
2 shared publications - Beng Heng Ang
Integrative Sciences and Engineering Programme, National University of Singapore
2 shared publications
Latest Publications
- Top Cited PapersOpen Access
- 18 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (12) , 1-33
Benchmarking LLMs for Unit Test Generation from Real-World Functions
Recently, large language models (LLMs) have shown great promise in automating unit test generation, significantly reducing the manual effort required by developers. To effectively evaluate the capabilities of LLMs in this domain, it is crucial to have a well-designed benchmark that accurately reflects real-world scenarios and mitigates common pitfalls. Existing LLM test generation benchmarks are limited by two critical drawbacks: data contamination and structurally simple function code. As a result, scientific conclusions drawn from empirical studies using these benchmarks may be compromised. The evidence presented may be biased due to contamination and may fail to generalize beyond toy programs due to structural simplicity. To address these problems, we introduce ULT (UnLeakedTestbench), a new benchmark specifically designed for function-level unit test generation from real-world Python functions. ULT is constructed through a multi-stage curation process that ensures high cyclomatic complexity and mitigates test case contamination. With 3,909 carefully selected function-level tasks, ULT provides a more realistic and challenging evaluation of LLMs’ test generation capabilities. We also provide PLT (PreLeakedTestbench), a pair of ULT benchmarks with leaked tests designed to enable a controlled analysis of memorization versus reasoning in test generation. Based on the two datasets, we conduct a large-scale empirical study involving 12 state-of-the-art LLMs, comparing their performance against established benchmarks. Our evaluation results demonstrate that ULT is significantly more challenging. For example, test cases generated by LLMs only achieve 41.32%, 45.10%, 30.22%, and 40.21% for accuracy, statement coverage, branch coverage, and mutation score on average for all LLMs, respectively. These results are substantially lower than the corresponding metrics on TestEval (91.79%, 92.18%, 82.04%, and 49.69%) and PLT (47.07%, 55.13%, 40.07%, and 50.80%). In addition, different from existing benchmarks, ULT shows a strong correlation between test generation performance and code generation performance. For example, the correlation coefficient between the coding ability and test generation performance ( \(Pass@1\) ) on ULT is 0.79 (p = 0.002), while it is only 0.56 (p = 0.059) and 0.52 (p = 0.080) on TestEval and PLT, respectively. This indicates that ULT more effectively measures the generalization ability of LLMs. We also make ULT and evaluation results publicly available to foster further research (To preserve the integrity of the benchmark and prevent test case leakage into future LLM training sets, we do not release the ground-truth tests. Instead, we provide the complete evaluation results for all models benchmarked in this article. This allows researchers to compare new models against our findings without compromising the benchmark). ULT is available at https://github.com/huangd1999/UnLeakedTestBench .- 18 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (12) , 1-33
Top Cited PapersOpen AccessBenchmarking LLMs for Unit Test Generation from Real-World Functions
Recently, large language models (LLMs) have shown great promise in automating unit test generation, significantly reducing the manual effort required by developers. To effectively evaluate the capabilities of LLMs in this domain, it is crucial to have a well-designed benchmark that accurately reflects real-world scenarios and mitigates common pitfalls. Existing LLM test generation benchmarks are limited by two critical drawbacks: data contamination and structurally simple function code. As a result, scientific conclusions drawn from empirical studies using these benchmarks may be compromised. The evidence presented may be biased due to contamination and may fail to generalize beyond toy programs due to structural simplicity. To address these problems, we introduce ULT (UnLeakedTestbench), a new benchmark specifically designed for function-level unit test generation from real-world Python functions. ULT is constructed through a multi-stage curation process that ensures high cyclomatic complexity and mitigates test case contamination. With 3,909 carefully selected function-level tasks, ULT provides a more realistic and challenging evaluation of LLMs’ test generation capabilities. We also provide PLT (PreLeakedTestbench), a pair of ULT benchmarks with leaked tests designed to enable a controlled analysis of memorization versus reasoning in test generation. Based on the two datasets, we conduct a large-scale empirical study involving 12 state-of-the-art LLMs, comparing their performance against established benchmarks. Our evaluation results demonstrate that ULT is significantly more challenging. For example, test cases generated by LLMs only achieve 41.32%, 45.10%, 30.22%, and 40.21% for accuracy, statement coverage, branch coverage, and mutation score on average for all LLMs, respectively. These results are substantially lower than the corresponding metrics on TestEval (91.79%, 92.18%, 82.04%, and 49.69%) and PLT (47.07%, 55.13%, 40.07%, and 50.80%). In addition, different from existing benchmarks, ULT shows a strong correlation between test generation performance and code generation performance. For example, the correlation coefficient between the coding ability and test generation performance ( \(Pass@1\) ) on ULT is 0.79 (p = 0.002), while it is only 0.56 (p = 0.059) and 0.52 (p = 0.080) on TestEval and PLT, respectively. This indicates that ULT more effectively measures the generalization ability of LLMs. We also make ULT and evaluation results publicly available to foster further research (To preserve the integrity of the benchmark and prevent test case leakage into future LLM training sets, we do not release the ground-truth tests. Instead, we provide the complete evaluation results for all models benchmarked in this article. This allows researchers to compare new models against our findings without compromising the benchmark). ULT is available at https://github.com/huangd1999/UnLeakedTestBench . - Open Access
- 6 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 8260-8267
Paper Espresso: From Paper Overload to Research Insight
The accelerating pace of scientific publishing makes it increasingly difficult for researchers to stay current. We present Paper Espresso , an open-source platform that automatically discovers, summarizes, and analyzes trending arXiv papers. The system uses large language models (LLMs) to generate structured summaries with topical labels and keywords, and provides multi-granularity trend analysis at daily, monthly, and lifecycle scales through LLM-driven topic consolidation. Over 36~months of continuous deployment, Paper Espresso has processed 13,388 papers and publicly released all structured metadata, revealing rich dynamics in the AI research landscape: a mid-2025 surge in reinforcement learning for LLM reasoning, non-saturating topic emergence (6,673 unique topics), and a positive correlation between topic novelty and community engagement (2.0× median upvotes for the most novel papers). The live demo and code are available at https://huggingface.co/spaces/Elfsong/Paper_Espresso.
- 6 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 8260-8267
Open AccessPaper Espresso: From Paper Overload to Research Insight
The accelerating pace of scientific publishing makes it increasingly difficult for researchers to stay current. We present Paper Espresso , an open-source platform that automatically discovers, summarizes, and analyzes trending arXiv papers. The system uses large language models (LLMs) to generate structured summaries with topical labels and keywords, and provides multi-granularity trend analysis at daily, monthly, and lifecycle scales through LLM-driven topic consolidation. Over 36~months of continuous deployment, Paper Espresso has processed 13,388 papers and publicly released all structured metadata, revealing rich dynamics in the AI research landscape: a mid-2025 surge in reinforcement learning for LLM reasoning, non-saturating topic emergence (6,673 unique topics), and a positive correlation between topic novelty and community engagement (2.0× median upvotes for the most novel papers). The live demo and code are available at https://huggingface.co/spaces/Elfsong/Paper_Espresso.
- Open Access
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 156-168
Measuring the Influence of Incorrect Code on Test Generation
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 156-168
Open AccessMeasuring the Influence of Incorrect Code on Test Generation
- Open Access
Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis
Test oracle generation in non-regression testing is a longstanding challenge in software engineering, where the goal is to produce oracles that can accurately determine whether a function under test (FUT) behaves as intended for a given input. In this paper, we introduce Nexus, a novel multi-agent framework to address this challenge. Nexus generates test oracles by leveraging a diverse set of specialized agents that synthesize test oracles through a structured process of deliberation, validation, and iterative self-refinement. During the deliberation phase, a panel of four specialist agents, each embodying a distinct testing philosophy, collaboratively critiques and refines an initial set of test oracles. Then, in the validation phase, Nexus generates a plausible candidate implementation of the FUT and executes the proposed oracles against it in a secure sandbox. For any oracle that fails this execution-based check, Nexus activates an automated selfrefinement loop, using the specific runtime error to debug and correct the oracle before re-validation. Our extensive evaluation on seven diverse benchmarks demonstrates that Nexus consistently and substantially outperforms state-of-theart baselines. For instance, Nexus improves the test-level oracle accuracy on the LiveCodeBench from 46.30% to 57.73% for GPT-4.1-Mini. The improved accuracy also significantly enhances downstream tasks: the bug detection rate of GPT4.1-Mini generated test oracles on HumanEval increases from 90.91% to 95.45% for Nexus compared to baselines, and the success rate of automated program repair improves from 35.23% to 69.32%.
Open AccessNexus: Execution-Grounded Multi-Agent Test Oracle Synthesis
Test oracle generation in non-regression testing is a longstanding challenge in software engineering, where the goal is to produce oracles that can accurately determine whether a function under test (FUT) behaves as intended for a given input. In this paper, we introduce Nexus, a novel multi-agent framework to address this challenge. Nexus generates test oracles by leveraging a diverse set of specialized agents that synthesize test oracles through a structured process of deliberation, validation, and iterative self-refinement. During the deliberation phase, a panel of four specialist agents, each embodying a distinct testing philosophy, collaboratively critiques and refines an initial set of test oracles. Then, in the validation phase, Nexus generates a plausible candidate implementation of the FUT and executes the proposed oracles against it in a secure sandbox. For any oracle that fails this execution-based check, Nexus activates an automated selfrefinement loop, using the specific runtime error to debug and correct the oracle before re-validation. Our extensive evaluation on seven diverse benchmarks demonstrates that Nexus consistently and substantially outperforms state-of-theart baselines. For instance, Nexus improves the test-level oracle accuracy on the LiveCodeBench from 46.30% to 57.73% for GPT-4.1-Mini. The improved accuracy also significantly enhances downstream tasks: the bug detection rate of GPT4.1-Mini generated test oracles on HumanEval increases from 90.91% to 95.45% for Nexus compared to baselines, and the success rate of automated program repair improves from 35.23% to 69.32%.
- Open Access
Benchmarking LLMs for Unit Test Generation from Real-World Functions
Recently, large language models (LLMs) have shown great promise in automating unit test generation, significantly reducing the manual effort required by developers. To effectively evaluate the capabilities of LLMs in this domain, it is crucial to have a well-designed benchmark that accurately reflects real-world scenarios and mitigates common pitfalls. Existing LLM test generation benchmarks are limited by two critical drawbacks: data contamination and structurally simple function code. As a result, we often cannot rely on the validity of scientific conclusions drawn from empirical studies using these limited benchmarks. The empirical evidence presented may be biased due to contamination and may fail to generalize beyond toy programs due to structural simplicity. To address these problems, we introduce ULT (UnLeakedTestbench), a new benchmark specifically designed for function-level unit test generation from real-world Python functions. ULT is constructed through a multi-stage curation process that ensures high cyclomatic complexity and mitigates test case contamination. With 3,909 carefully selected function-level tasks, ULT provides a more realistic and challenging evaluation of LLMs' test generation capabilities. We also provide PLT (PreLeakedTestbench), a pair benchmark of ULT with leaked tests designed to enable a controlled analysis of memorization versus reasoning in test generation. Our evaluation results demonstrate that ULT is significantly more challenging. For example, test cases generated by LLMs only achieve 41.32\%, 45.10\%, 30.22\%, and 40.21\% for accuracy, statement coverage, branch coverage, and mutation score on average for all LLMs, respectively. These results are substantially lower than the corresponding metrics on TestEval (91.79\%, 92.18\%, 82.04\%, and 49.69\%) and PLT (47.07\%, 55.13\%, 40.07\%, and 50.80\%).
Open AccessBenchmarking LLMs for Unit Test Generation from Real-World Functions
Recently, large language models (LLMs) have shown great promise in automating unit test generation, significantly reducing the manual effort required by developers. To effectively evaluate the capabilities of LLMs in this domain, it is crucial to have a well-designed benchmark that accurately reflects real-world scenarios and mitigates common pitfalls. Existing LLM test generation benchmarks are limited by two critical drawbacks: data contamination and structurally simple function code. As a result, we often cannot rely on the validity of scientific conclusions drawn from empirical studies using these limited benchmarks. The empirical evidence presented may be biased due to contamination and may fail to generalize beyond toy programs due to structural simplicity. To address these problems, we introduce ULT (UnLeakedTestbench), a new benchmark specifically designed for function-level unit test generation from real-world Python functions. ULT is constructed through a multi-stage curation process that ensures high cyclomatic complexity and mitigates test case contamination. With 3,909 carefully selected function-level tasks, ULT provides a more realistic and challenging evaluation of LLMs' test generation capabilities. We also provide PLT (PreLeakedTestbench), a pair benchmark of ULT with leaked tests designed to enable a controlled analysis of memorization versus reasoning in test generation. Our evaluation results demonstrate that ULT is significantly more challenging. For example, test cases generated by LLMs only achieve 41.32\%, 45.10\%, 30.22\%, and 40.21\% for accuracy, statement coverage, branch coverage, and mutation score on average for all LLMs, respectively. These results are substantially lower than the corresponding metrics on TestEval (91.79\%, 92.18\%, 82.04\%, and 49.69\%) and PLT (47.07\%, 55.13\%, 40.07\%, and 50.80\%).
- Open Access
- 11 April 2025
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (23) , 24230-24238
Towards Verifiable Text Generation with Generative Agent
Text generation with citations makes it easy to verify the factuality of Large Language Models’ (LLMs) generations. Existing one-step generation studies expose distinct shortages in answer refinement and in-context demonstration matching. In light of these challenges, we propose R2-MGA, a Retrieval and Reflection Memory-augmented Generative Agent. Specifically, it first retrieves the memory bank to obtain the best-matched memory snippet, then reflects the retrieved snippet as a reasoning rationale, next combines the snippet and the rationale as the best-matched in-context demonstration. Additionally, it is capable of in-depth answer refinement with two specifically designed modules. We evaluate R2-MGA across five LLMs on the ALCE benchmark. The results reveal R2-MGA’ exceptional capabilities in text generation with citations. In particular, compared to the selected baselines, it delivers up to +58.8% and +154.7% relative performance gains on answer correctness and citation quality, respectively. Extensive analyses strongly support the motivations of R2-MGA.
- 11 April 2025
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (23) , 24230-24238
Open AccessTowards Verifiable Text Generation with Generative Agent
Text generation with citations makes it easy to verify the factuality of Large Language Models’ (LLMs) generations. Existing one-step generation studies expose distinct shortages in answer refinement and in-context demonstration matching. In light of these challenges, we propose R2-MGA, a Retrieval and Reflection Memory-augmented Generative Agent. Specifically, it first retrieves the memory bank to obtain the best-matched memory snippet, then reflects the retrieved snippet as a reasoning rationale, next combines the snippet and the rationale as the best-matched in-context demonstration. Additionally, it is capable of in-depth answer refinement with two specifically designed modules. We evaluate R2-MGA across five LLMs on the ALCE benchmark. The results reveal R2-MGA’ exceptional capabilities in text generation with citations. In particular, compared to the selected baselines, it delivers up to +58.8% and +154.7% relative performance gains on answer correctness and citation quality, respectively. Extensive analyses strongly support the motivations of R2-MGA.
- Open Access
- 31 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 1004-1006
Curriculum Demonstration Selection for In-Context Learning
- 31 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 1004-1006
Open AccessCurriculum Demonstration Selection for In-Context Learning
- Open Access
- 7 February 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 27 (1) , e59524
Unraveling Online Mental Health Through the Lens of Early Maladaptive Schemas: AI-Enabled Content Analysis of Online Mental Health Communities
Background: Early maladaptive schemas (EMSs) are pervasive, self-defeating patterns of thoughts and emotions underlying most mental health problems and are central in schema therapy. However, the characteristics of EMSs vary across demographics, and despite the growing use of online mental health communities (OMHCs), how EMSs manifest in these online support-seeking environments remains unclear. Understanding these characteristics could inform the design of more effective interventions powered by artificial intelligence to address online support seekers’ unique therapeutic needs. Objective: We aimed to uncover associations between EMSs and mental health problems within OMHCs and examine features of EMSs as they are reflected in OMHCs. Methods: We curated a dataset of 29,329 posts from widely accessed OMHCs, labeling each with relevant schemas and mental health problems. To identify associations, we conducted chi-square tests of independence and calculated odds ratios (ORs) with the dataset. In addition, we developed a novel group-level case conceptualization technique, leveraging GPT-4 to extract features of EMSs from OMHC texts across key schema therapy dimensions, such as schema triggers and coping responses. Results: Several associations were identified between EMSs and mental health problems, reflecting how EMSs manifest in online support-seeking contexts. Anxiety-related problems typically highlighted vulnerability to harm or illness (OR 5.64, 95% CI 5.34-5.96; P<.001), while depression-related problems emphasized unmet interpersonal needs, such as social isolation (OR 3.18, 95% CI 3.02-3.34; P<.001). Conversely, problems with eating disorders mostly exemplified negative self-perception and emotional inhibition (OR 1.89, 95% CI 1.45-2.46; P<.001). Personality disorders reflected themes of subjugation (OR 2.51, 95% CI 1.86-3.39; P<.001), while posttraumatic stress disorder problems involved distressing experiences and mistrust (OR 5.04, 95% CI 4.49-5.66; P<.001). Substance use disorder problems reflected negative self-perception of failure to achieve (OR 1.83, 95% CI 1.35-2.49; P<.001). Depression, personality disorders, and posttraumatic stress disorder were also associated with 12, 9, and 7 EMSs, respectively, emphasizing their complexities and the need for more comprehensive interventions. In contrast, anxiety, eating disorder, and substance use disorder were related to only 2 to 3 EMSs, suggesting that these problems are better addressed through targeted interventions. In addition, the EMS features extracted from our dataset averaged 13.27 (SD 3.05) negative features per schema, with 2.65 (SD 1.07) features per dimension, as supported by existing literature. Conclusions: We uncovered various associations between EMSs and mental health problems among online support seekers, highlighting the prominence of specific EMSs in each problem and the unique complexities of each problem in terms of EMSs. We also identified EMS features as expressed by support seekers in OMHCs, reinforcing the relevance of EMSs in these online support-seeking contexts. These insights are valuable for understanding how EMS are characterized in OMHCs and can inform the development of more effective artificial intelligence–powered tools to enhance support on these platforms.
- 7 February 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 27 (1) , e59524
Open AccessUnraveling Online Mental Health Through the Lens of Early Maladaptive Schemas: AI-Enabled Content Analysis of Online Mental Health Communities
Background: Early maladaptive schemas (EMSs) are pervasive, self-defeating patterns of thoughts and emotions underlying most mental health problems and are central in schema therapy. However, the characteristics of EMSs vary across demographics, and despite the growing use of online mental health communities (OMHCs), how EMSs manifest in these online support-seeking environments remains unclear. Understanding these characteristics could inform the design of more effective interventions powered by artificial intelligence to address online support seekers’ unique therapeutic needs. Objective: We aimed to uncover associations between EMSs and mental health problems within OMHCs and examine features of EMSs as they are reflected in OMHCs. Methods: We curated a dataset of 29,329 posts from widely accessed OMHCs, labeling each with relevant schemas and mental health problems. To identify associations, we conducted chi-square tests of independence and calculated odds ratios (ORs) with the dataset. In addition, we developed a novel group-level case conceptualization technique, leveraging GPT-4 to extract features of EMSs from OMHC texts across key schema therapy dimensions, such as schema triggers and coping responses. Results: Several associations were identified between EMSs and mental health problems, reflecting how EMSs manifest in online support-seeking contexts. Anxiety-related problems typically highlighted vulnerability to harm or illness (OR 5.64, 95% CI 5.34-5.96; P<.001), while depression-related problems emphasized unmet interpersonal needs, such as social isolation (OR 3.18, 95% CI 3.02-3.34; P<.001). Conversely, problems with eating disorders mostly exemplified negative self-perception and emotional inhibition (OR 1.89, 95% CI 1.45-2.46; P<.001). Personality disorders reflected themes of subjugation (OR 2.51, 95% CI 1.86-3.39; P<.001), while posttraumatic stress disorder problems involved distressing experiences and mistrust (OR 5.04, 95% CI 4.49-5.66; P<.001). Substance use disorder problems reflected negative self-perception of failure to achieve (OR 1.83, 95% CI 1.35-2.49; P<.001). Depression, personality disorders, and posttraumatic stress disorder were also associated with 12, 9, and 7 EMSs, respectively, emphasizing their complexities and the need for more comprehensive interventions. In contrast, anxiety, eating disorder, and substance use disorder were related to only 2 to 3 EMSs, suggesting that these problems are better addressed through targeted interventions. In addition, the EMS features extracted from our dataset averaged 13.27 (SD 3.05) negative features per schema, with 2.65 (SD 1.07) features per dimension, as supported by existing literature. Conclusions: We uncovered various associations between EMSs and mental health problems among online support seekers, highlighting the prominence of specific EMSs in each problem and the unique complexities of each problem in terms of EMSs. We also identified EMS features as expressed by support seekers in OMHCs, reinforcing the relevance of EMSs in these online support-seeking contexts. These insights are valuable for understanding how EMS are characterized in OMHCs and can inform the development of more effective artificial intelligence–powered tools to enhance support on these platforms.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 1092-1103
On Assigning Product and Software Codes to Customer Service Requests with Large Language Models
Sujatha Das Gollapalli, Mouad Hakam, Mingzhe Du, See-Kiong Ng, Mohammed Hamzeh. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 1092-1103
Open AccessOn Assigning Product and Software Codes to Customer Service Requests with Large Language Models
Sujatha Das Gollapalli, Mouad Hakam, Mingzhe Du, See-Kiong Ng, Mohammed Hamzeh. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2025.
- Open Access
- 16 October 2024
- Published by IOS Press
- p. 2556-2563
Counseling Responses for Mental Health Forum Questions with Early Maladaptive Schema Prediction
State-of-the-art Large Language Models (LLMs) have shown remarkable capabilities for general Question Answering (QA) tasks. However, their practical use for answering mental health questions has been limited due to the missing link between LLM-generated answer responses and well-established theories and guiding principles from Psychology and Counseling. We present a first step in this direction with STeer, an AI-based method that supports Schema Therapy-enabled responses for mental health questions on community QA forums. STeer uses Early Maladaptive Schemas (EMSs), a fundamental concept from Schema Therapy that characterizes “self-defeating, unhealthy patterns of thought and behavior” in individuals, to effectively prompt state-of-the-art LLMs to generate specific, theoretically-grounded, empathetic counseling responses to mental health questions. We present EMSRank, a novel method based on the Personalized PageRank algorithm, to automatically predict the EMSs from mental health forum question texts. We show that EMSRank is computationally scalable and can be further combined with textual entailment to obtain high precision, explainable EMS labels for mental health forum questions. To address the current lack of annotated datasets, we also leveraged on EMSRank to create a first-of-its-kind, large dataset of about 23K EMS-annotated mental health questions from three diverse, currently operating, peer-support community forums for mental health. With the global rise in mental health issues, our work is a timely step towards enabling the use of AI-based assistive tools for counseling support on mental health community forums.
- 16 October 2024
- Published by IOS Press
- p. 2556-2563
Open AccessCounseling Responses for Mental Health Forum Questions with Early Maladaptive Schema Prediction
State-of-the-art Large Language Models (LLMs) have shown remarkable capabilities for general Question Answering (QA) tasks. However, their practical use for answering mental health questions has been limited due to the missing link between LLM-generated answer responses and well-established theories and guiding principles from Psychology and Counseling. We present a first step in this direction with STeer, an AI-based method that supports Schema Therapy-enabled responses for mental health questions on community QA forums. STeer uses Early Maladaptive Schemas (EMSs), a fundamental concept from Schema Therapy that characterizes “self-defeating, unhealthy patterns of thought and behavior” in individuals, to effectively prompt state-of-the-art LLMs to generate specific, theoretically-grounded, empathetic counseling responses to mental health questions. We present EMSRank, a novel method based on the Personalized PageRank algorithm, to automatically predict the EMSs from mental health forum question texts. We show that EMSRank is computationally scalable and can be further combined with textual entailment to obtain high precision, explainable EMS labels for mental health forum questions. To address the current lack of annotated datasets, we also leveraged on EMSRank to create a first-of-its-kind, large dataset of about 23K EMS-annotated mental health questions from three diverse, currently operating, peer-support community forums for mental health. With the global rise in mental health issues, our work is a timely step towards enabling the use of AI-based assistive tools for counseling support on mental health community forums.