Huadong Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Huadong Wang has a total of 85 co-authors. Here are the top co-authors by number of shared publications:
- Zhiyuan Liu
Tsinghua University
4 shared publications - Jiabin Liu
School of Information and Electronics, Beijing Institute of Technology, Beijing, China
3 shared publications - Yong Shi
School of Economics and Management, University of Chinese Academy of Sciences, Beijing 100190, China
3 shared publications - Bo Wang
School of Information Technology and Management, University of International Business and Economics, Beijing, China
2 shared publications - Lingfeng Niu
School of Economics and Management, University of Chinese Academy of Sciences, Beijing, China
2 shared publications Maosong Sun
Dept. of Comp. Sci. & Tech Institute for AI Tsinghua University Beijing China
2 shared publications
Latest Publications
- 23 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (11) , 19951-19963
Progressive Training for Learning From Label Proportions
Learning from label proportions (LLPs), which aims to learn an instance-level classifier using proportion-based grouped training data, has garnered increasing attention in the field of machine learning. Existing deep learning-based LLP methods employ end-to-end pipelines to derive proportional loss functions via the Kullback–Leibler (KL) divergence between bag-level prior and posterior class distributions. However, the optimal solutions of these methods often struggle to conform to the given proportions, inevitably leading to degradation in the final classification performance. In this article, we address this issue by proposing a novel progressive training method for LLP, termed PT-LLP, which strives to meet the proportion constraints from the bag level to the instance level. Specifically, we first train a model by using the existing KL-divergence-based LLP methods that are consistent with bag-level proportion information. Then, we impose additional constraints on strict proportion consistency to the classifier to further move toward a more ideal solution by reformulating it as a constrained optimization problem, which can be efficiently solved using optimal transport (OT) algorithms. In particular, the knowledge distillation is employed as a transition stage to transfer the bag-level information to the instance level using a teacher–student framework. Finally, our framework is model-agnostic and demonstrates significant performance improvements through extensive experiments on different datasets when incorporated into other deep LLP methods as the first training stage.
- 23 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (11) , 19951-19963
Progressive Training for Learning From Label Proportions
Learning from label proportions (LLPs), which aims to learn an instance-level classifier using proportion-based grouped training data, has garnered increasing attention in the field of machine learning. Existing deep learning-based LLP methods employ end-to-end pipelines to derive proportional loss functions via the Kullback–Leibler (KL) divergence between bag-level prior and posterior class distributions. However, the optimal solutions of these methods often struggle to conform to the given proportions, inevitably leading to degradation in the final classification performance. In this article, we address this issue by proposing a novel progressive training method for LLP, termed PT-LLP, which strives to meet the proportion constraints from the bag level to the instance level. Specifically, we first train a model by using the existing KL-divergence-based LLP methods that are consistent with bag-level proportion information. Then, we impose additional constraints on strict proportion consistency to the classifier to further move toward a more ideal solution by reformulating it as a constrained optimization problem, which can be efficiently solved using optimal transport (OT) algorithms. In particular, the knowledge distillation is employed as a transition stage to transfer the bag-level information to the instance level using a teacher–student framework. Finally, our framework is model-agnostic and demonstrates significant performance improvements through extensive experiments on different datasets when incorporated into other deep LLP methods as the first training stage.
- Top Cited PapersOpen Access
- 24 December 2024
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 57 (4) , 1-40
Tool Learning with Foundation Models
Humans possess an extraordinary ability to create and utilize tools. With the advent of foundation models, artificial intelligence systems have the potential to be equally adept in tool use as humans. This paradigm, which is dubbed as tool learning with foundation models , combines the strengths of tools and foundation models to achieve enhanced accuracy, efficiency, and automation in problem-solving. This article presents a systematic investigation and comprehensive review of tool learning. We first introduce the background of tool learning, including its cognitive origins, the paradigm shift of foundation models, and the complementary roles of tools and models. Then we recapitulate existing tool learning research and formulate a general framework: starting from understanding the user instruction, models should learn to decompose a complex task into several subtasks, dynamically adjust their plan through reasoning, and effectively conquer each sub-task by selecting appropriate tools. We also discuss how to train models for improved tool-use capabilities and facilitate generalization in tool learning. Finally, we discuss several open problems that require further investigation, such as ensuring trustworthy tool use, enabling tool creation with foundation models, and addressing personalization challenges. Overall, we hope this article could inspire future research in integrating tools with foundation models.
- 24 December 2024
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 57 (4) , 1-40
Top Cited PapersOpen AccessTool Learning with Foundation Models
Humans possess an extraordinary ability to create and utilize tools. With the advent of foundation models, artificial intelligence systems have the potential to be equally adept in tool use as humans. This paradigm, which is dubbed as tool learning with foundation models , combines the strengths of tools and foundation models to achieve enhanced accuracy, efficiency, and automation in problem-solving. This article presents a systematic investigation and comprehensive review of tool learning. We first introduce the background of tool learning, including its cognitive origins, the paradigm shift of foundation models, and the complementary roles of tools and models. Then we recapitulate existing tool learning research and formulate a general framework: starting from understanding the user instruction, models should learn to decompose a complex task into several subtasks, dynamically adjust their plan through reasoning, and effectively conquer each sub-task by selecting appropriate tools. We also discuss how to train models for improved tool-use capabilities and facilitate generalization in tool learning. Finally, we discuss several open problems that require further investigation, such as ensuring trustworthy tool use, enabling tool creation with foundation models, and addressing personalization challenges. Overall, we hope this article could inspire future research in integrating tools with foundation models.
- 23 January 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (2) , 2675-2689
Self-Supervised Random Forest on Transformed Distribution for Anomaly Detection
Anomaly detection, the task of differentiating abnormal data points from normal ones, presents a significant challenge in the realm of machine learning. Numerous strategies have been proposed to tackle this task, with classification-based methods, specifically those utilizing a self-supervised approach via random affine transformations (RATs), demonstrating remarkable performance on both image and non-image data. However, these methods encounter a notable bottleneck, the overlap of constructed labeled datasets across categories, which hampers the subsequent classifiers’ ability to detect anomalies. Consequently, the creation of an effective data distribution becomes the pivotal factor for success. In this article, we introduce a model called “self-supervised forest (sForest)”, which leverages the random Fourier transform (RFT) and random orthogonal rotations to craft a controlled data distribution. Our model utilizes the RFT to map input data into a new feature space. With this transformed data, we create a self-labeled training dataset using random orthogonal rotations. We theoretically prove that the data distribution formulated by our methodology is more stable compared to one derived from RATs. We then use the self-labeled dataset in a random forest (RF) classifier to distinguish between normal and anomalous data points. Comprehensive experiments conducted on both real and artificial datasets illustrate that sForest outperforms other anomaly detection methods, including distance-based, kernel-based, forest-based, and network-based benchmarks.
- 23 January 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (2) , 2675-2689
Self-Supervised Random Forest on Transformed Distribution for Anomaly Detection
Anomaly detection, the task of differentiating abnormal data points from normal ones, presents a significant challenge in the realm of machine learning. Numerous strategies have been proposed to tackle this task, with classification-based methods, specifically those utilizing a self-supervised approach via random affine transformations (RATs), demonstrating remarkable performance on both image and non-image data. However, these methods encounter a notable bottleneck, the overlap of constructed labeled datasets across categories, which hampers the subsequent classifiers’ ability to detect anomalies. Consequently, the creation of an effective data distribution becomes the pivotal factor for success. In this article, we introduce a model called “self-supervised forest (sForest)”, which leverages the random Fourier transform (RFT) and random orthogonal rotations to craft a controlled data distribution. Our model utilizes the RFT to map input data into a new feature space. With this transformed data, we create a self-labeled training dataset using random orthogonal rotations. We theoretically prove that the data distribution formulated by our methodology is more stable compared to one derived from RATs. We then use the self-labeled dataset in a random forest (RF) classifier to distinguish between normal and anomalous data points. Comprehensive experiments conducted on both real and artificial datasets illustrate that sForest outperforms other anomaly detection methods, including distance-based, kernel-based, forest-based, and network-based benchmarks.
- Open Access
Exploring Format Consistency for Instruction Tuning
Instruction tuning has emerged as a promising approach to enhancing large language models in following human instructions. It is shown that increasing the diversity and number of instructions in the training data can consistently enhance generalization performance, which facilitates a recent endeavor to collect various instructions and integrate existing instruction tuning datasets into larger collections. However, different users have their unique ways of expressing instructions, and there often exist variations across different datasets in the instruction styles and formats, i.e., format inconsistency. In this work, we propose a framework named Unified Instruction Tuning (UIT), which calls OpenAI APIs for automatic format transfer among different instruction tuning datasets such as PromptSource, FLAN and CrossFit. With the framework, we (1) demonstrate the necessity of maintaining format consistency in instruction tuning; (2) improve the generalization performance on unseen instructions on T5-LM-xl; (3) provide a novel perplexity-based denoising method to reduce the noise of automatic format transfer to make the UIT framework more practical and a smaller offline model based on GPT-J that achieves comparable format transfer capability to OpenAI APIs to reduce costs in practice. Further analysis regarding variations of targeted formats and other effects is intended.
Open AccessExploring Format Consistency for Instruction Tuning
Instruction tuning has emerged as a promising approach to enhancing large language models in following human instructions. It is shown that increasing the diversity and number of instructions in the training data can consistently enhance generalization performance, which facilitates a recent endeavor to collect various instructions and integrate existing instruction tuning datasets into larger collections. However, different users have their unique ways of expressing instructions, and there often exist variations across different datasets in the instruction styles and formats, i.e., format inconsistency. In this work, we propose a framework named Unified Instruction Tuning (UIT), which calls OpenAI APIs for automatic format transfer among different instruction tuning datasets such as PromptSource, FLAN and CrossFit. With the framework, we (1) demonstrate the necessity of maintaining format consistency in instruction tuning; (2) improve the generalization performance on unseen instructions on T5-LM-xl; (3) provide a novel perplexity-based denoising method to reduce the noise of automatic format transfer to make the UIT framework more practical and a smaller offline model based on GPT-J that achieves comparable format transfer capability to OpenAI APIs to reduce costs in practice. Further analysis regarding variations of targeted formats and other effects is intended.
- Open Access
Won't Get Fooled Again: Answering Questions with False Premises
Pre-trained language models (PLMs) have shown unprecedented potential in various fields, especially as the backbones for question-answering (QA) systems. However, they tend to be easily deceived by tricky questions such as "How many eyes does the sun have?". Such frailties of PLMs often allude to the lack of knowledge within them. In this paper, we find that the PLMs already possess the knowledge required to rebut such questions, and the key is how to activate the knowledge. To systematize this observation, we investigate the PLMs' responses to one kind of tricky questions, i.e., the false premises questions (FPQs). We annotate a FalseQA dataset containing 2365 human-written FPQs, with the corresponding explanations for the false premises and the revised true premise questions. Using FalseQA, we discover that PLMs are capable of discriminating FPQs by fine-tuning on moderate numbers (e.g., 256) of examples. PLMs also generate reasonable explanations for the false premise, which serve as rebuttals. Further replaying a few general questions during training allows PLMs to excel on FPQs and general questions simultaneously. Our work suggests that once the rebuttal ability is stimulated, knowledge inside the PLMs can be effectively utilized to handle FPQs, which incentivizes the research on PLM-based QA systems.
Open AccessWon't Get Fooled Again: Answering Questions with False Premises
Pre-trained language models (PLMs) have shown unprecedented potential in various fields, especially as the backbones for question-answering (QA) systems. However, they tend to be easily deceived by tricky questions such as "How many eyes does the sun have?". Such frailties of PLMs often allude to the lack of knowledge within them. In this paper, we find that the PLMs already possess the knowledge required to rebut such questions, and the key is how to activate the knowledge. To systematize this observation, we investigate the PLMs' responses to one kind of tricky questions, i.e., the false premises questions (FPQs). We annotate a FalseQA dataset containing 2365 human-written FPQs, with the corresponding explanations for the false premises and the revised true premise questions. Using FalseQA, we discover that PLMs are capable of discriminating FPQs by fine-tuning on moderate numbers (e.g., 256) of examples. PLMs also generate reasonable explanations for the false premise, which serve as rebuttals. Further replaying a few general questions during training allows PLMs to excel on FPQs and general questions simultaneously. Our work suggests that once the rebuttal ability is stimulated, knowledge inside the PLMs can be effectively utilized to handle FPQs, which incentivizes the research on PLM-based QA systems.
- 14 June 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 61 (01962892) , 1-19
Land Use and Land Cover Mapping in China Using Multimodal Fine-Grained Dual Network
With the advancement of geo-systems and the increased availability of satellite data, a plethora of Land-Use and Land-Cover (LULC) products have been developed. The existing LULC products primarily relied on time-series imagery to classify land by pixel-based classifiers, allowing for local analysis and accurate boundary detection. However, the advent of deep learning has shifted towards the use of patch-based CNN models for generating land cover maps. In this paper, (1) we create a training dataset for China using a voting strategy based on three off-the-shelf available LULC products, avoiding the labor-intensive manual annotation. (2) We design a novel CNN-based model for LULC task, called Multi-modal Fine-grained Dual Network (dubbed as Dual-Net), which takes dual-date images to generate final maps, and reduces the need for gap-free temporal sequences or separate cloud detection. To leverage the correlation between location, date, and category, we embed multi-modal information (dates and geo-locations) to the model. Further, by incorporating low-level constraints and using pseudo-label refinement, we continually improve the performance and achieve more refined segmentation. (3) Due to the lack of a suitable validation dataset for China, we create a new validation dataset called China Sentinel2 Validation Dataset (CSVD) by manually annotating 733 finely labeled images of 1024 × 1024 pixels of China-specific Sentinel2 data. (4) Extensive experiments demonstrate that our model outperforms existing LULC products and produces more fine-grained segmentation results comparable to other patch-based products. Finally, we release annual LULC maps for China in 2020-2022 and also make our model accessible online for real-time results export.
- 14 June 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 61 (01962892) , 1-19
Land Use and Land Cover Mapping in China Using Multimodal Fine-Grained Dual Network
With the advancement of geo-systems and the increased availability of satellite data, a plethora of Land-Use and Land-Cover (LULC) products have been developed. The existing LULC products primarily relied on time-series imagery to classify land by pixel-based classifiers, allowing for local analysis and accurate boundary detection. However, the advent of deep learning has shifted towards the use of patch-based CNN models for generating land cover maps. In this paper, (1) we create a training dataset for China using a voting strategy based on three off-the-shelf available LULC products, avoiding the labor-intensive manual annotation. (2) We design a novel CNN-based model for LULC task, called Multi-modal Fine-grained Dual Network (dubbed as Dual-Net), which takes dual-date images to generate final maps, and reduces the need for gap-free temporal sequences or separate cloud detection. To leverage the correlation between location, date, and category, we embed multi-modal information (dates and geo-locations) to the model. Further, by incorporating low-level constraints and using pseudo-label refinement, we continually improve the performance and achieve more refined segmentation. (3) Due to the lack of a suitable validation dataset for China, we create a new validation dataset called China Sentinel2 Validation Dataset (CSVD) by manually annotating 733 finely labeled images of 1024 × 1024 pixels of China-specific Sentinel2 data. (4) Extensive experiments demonstrate that our model outperforms existing LULC products and produces more fine-grained segmentation results comparable to other patch-based products. Finally, we release annual LULC maps for China in 2020-2022 and also make our model accessible online for real-time results export.
- 30 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 30 (23299290) , 2754-2767
Improved Transformer With Multi-Head Dense Collaboration
Recently, the attention mechanism boosts the performance of many neural network models in Natural Language Processing (NLP). Among the various attention mechanisms, Multi-Head Attention (MHA) is a powerful and popular variant. MHA helps the model to attend to different feature subspaces independently which is an essential component of Transformer. Despite its success, we conjecture that the different heads of the existing MHA may not collaborate properly. To validate this assumption and further improve the performance of Transformer, we study the collaboration problem for MHA in this paper. First, we propose the Single-Layer Collaboration (SLC) mechanism to help each attention head improve its attention distribution based on the feedback of other heads. Furthermore, we extend SLC to the cross-layer Multi-Head Dense Collaboration (MHDC) mechanism. MHDC helps each MHA layer learn the attention distributions considering the knowledge from the other MHA layers. Both SLC and MHDC are implemented as lightweight modules with very few additional parameters. When equipped with these modules, our new framework, i.e., Collaborative TransFormer ( CollFormer ), significantly outperforms the vanilla Transformer on a range of NLP tasks, including machine translation, sentence semantic relatedness, natural language inference, sentence classification, and reading comprehension. Besides, we also carry out extensive quantitative experiments to analyze the properties of the MHDC in different settings. The experimental results validate the effectiveness and universality of MHDC as well as CollFormer .
- 30 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 30 (23299290) , 2754-2767
Improved Transformer With Multi-Head Dense Collaboration
Recently, the attention mechanism boosts the performance of many neural network models in Natural Language Processing (NLP). Among the various attention mechanisms, Multi-Head Attention (MHA) is a powerful and popular variant. MHA helps the model to attend to different feature subspaces independently which is an essential component of Transformer. Despite its success, we conjecture that the different heads of the existing MHA may not collaborate properly. To validate this assumption and further improve the performance of Transformer, we study the collaboration problem for MHA in this paper. First, we propose the Single-Layer Collaboration (SLC) mechanism to help each attention head improve its attention distribution based on the feedback of other heads. Furthermore, we extend SLC to the cross-layer Multi-Head Dense Collaboration (MHDC) mechanism. MHDC helps each MHA layer learn the attention distributions considering the knowledge from the other MHA layers. Both SLC and MHDC are implemented as lightweight modules with very few additional parameters. When equipped with these modules, our new framework, i.e., Collaborative TransFormer ( CollFormer ), significantly outperforms the vanilla Transformer on a range of NLP tasks, including machine translation, sentence semantic relatedness, natural language inference, sentence classification, and reading comprehension. Besides, we also carry out extensive quantitative experiments to analyze the properties of the MHDC in different settings. The experimental results validate the effectiveness and universality of MHDC as well as CollFormer .
- Open Access
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification
Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification. Particularly, previous studies suggest that prompt-tuning has remarkable superiority in the low-data scenario over the generic fine-tuning methods with extra classifiers. The core idea of prompt-tuning is to insert text pieces, i.e., template, to the input and transform a classification problem into a masked language modeling problem, where a crucial step is to construct a projection, i.e., verbalizer, between a label space and a label word space. A verbalizer is usually handcrafted or searched by gradient descent, which may lack coverage and bring considerable bias and high variances to the results. In this work, we focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt-tuning (KPT), to improve and stabilize prompt-tuning. Specifically, we expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Extensive experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.
Open AccessKnowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification
Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification. Particularly, previous studies suggest that prompt-tuning has remarkable superiority in the low-data scenario over the generic fine-tuning methods with extra classifiers. The core idea of prompt-tuning is to insert text pieces, i.e., template, to the input and transform a classification problem into a masked language modeling problem, where a crucial step is to construct a projection, i.e., verbalizer, between a label space and a label word space. A verbalizer is usually handcrafted or searched by gradient descent, which may lack coverage and bring considerable bias and high variances to the results. In this work, we focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt-tuning (KPT), to improve and stabilize prompt-tuning. Specifically, we expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Extensive experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.
- 21 February 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (11) , 8377-8388
LLP-GAN: A GAN-Based Algorithm for Learning From Label Proportions
Learning from label proportions (LLP) is a widespread and important learning paradigm: only the bag-level proportional information of the grouped training instances is available for the classification task, instead of the instance-level labels in the fully supervised scenario. As a result, LLP is a typical weakly supervised learning protocol and commonly exists in privacy protection circumstances due to the sensitivity in label information for real-world applications. In general, it is less laborious and more efficient to collect label proportions as the bag-level supervised information than the instance-level one. However, the hint for learning the discriminative feature representation is also limited as a less informative signal directly associated with the labels is provided, thus deteriorating the performance of the final instance-level classifier. In this article, delving into the label proportions, we bypass this weak supervision by leveraging generative adversarial networks (GANs) to derive an effective algorithm LLP-GAN. Endowed with an end-to-end structure, LLP-GAN performs approximation in the light of an adversarial learning mechanism without imposing restricted assumptions on distribution. Accordingly, the final instance-level classifier can be directly induced upon the discriminator with minor modification. Under mild assumptions, we give the explicit generative representation and prove the global optimality for LLP-GAN. In addition, compared with existing methods, our work empowers LLP solvers with desirable scalability inheriting from deep models. Extensive experiments on benchmark datasets and a real-world application demonstrate the vivid advantages of the proposed approach.
- 21 February 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (11) , 8377-8388
LLP-GAN: A GAN-Based Algorithm for Learning From Label Proportions
Learning from label proportions (LLP) is a widespread and important learning paradigm: only the bag-level proportional information of the grouped training instances is available for the classification task, instead of the instance-level labels in the fully supervised scenario. As a result, LLP is a typical weakly supervised learning protocol and commonly exists in privacy protection circumstances due to the sensitivity in label information for real-world applications. In general, it is less laborious and more efficient to collect label proportions as the bag-level supervised information than the instance-level one. However, the hint for learning the discriminative feature representation is also limited as a less informative signal directly associated with the labels is provided, thus deteriorating the performance of the final instance-level classifier. In this article, delving into the label proportions, we bypass this weak supervision by leveraging generative adversarial networks (GANs) to derive an effective algorithm LLP-GAN. Endowed with an end-to-end structure, LLP-GAN performs approximation in the light of an adversarial learning mechanism without imposing restricted assumptions on distribution. Accordingly, the final instance-level classifier can be directly induced upon the discriminator with minor modification. Under mild assumptions, we give the explicit generative representation and prove the global optimality for LLP-GAN. In addition, compared with existing methods, our work empowers LLP solvers with desirable scalability inheriting from deep models. Extensive experiments on benchmark datasets and a real-world application demonstrate the vivid advantages of the proposed approach.
- Open Access
- 18 July 2018
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 20 (7) , e240
Evaluating Doctor Performance: Ordinal Regression-Based Approach
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Doctor performance evaluation (DPE) is an important task in mHealth, which aims to evaluate the overall quality of online diagnosis and patient outcomes so that high customer satisfaction and loyalty can be attained. However in reality most customers trend not to give ratings to doctors’ performance. Therefore it is imperative to develop a model to make DPE automatically. When making auto-evaluation of doctors’ performance, we expect to rate the doctors’ performance into a score label that is as close as possible to the true one. Objective: This study aims to do DPE automatically from online textual consultation contents between doctors and customers by a novel machine learning method. Methods: We propose a solution that models DPE as an ordinal regression problem. In doing so, a combined support vector machine and ordinal partitioning model (called SVMOP for short), along with an innovative prediction function is developed to capture the ordering labels hidden in preferences over DPE. Specifically, when conducting the feature engineering, in addition to the basic text features, eight handcrafted features extracted from over 70,000 medical entries, are added and then further boosted by Gradient Boosting Decision Tree. Results: Real data sets from one of the largest mobile doctor-patient communication platforms in China are used in our study. In according with the statistics, 64% of data on mHealth platforms lack the evaluation labels from customers. Experimental results reveal that our approach can well support doctor performance evaluation automatically. Specifically, compared with other auto-evaluation models, SVMOP improves mean absolute error (MAE) by 0.1, mean square error (MSE) by 0.5, pairwise accuracy (PAcc) by 5%; the handcrafted features that we suggest, improve MAE by 0.1, MSE by 0.2, PAcc by 3%. After boosting, the performance can be further improved. Furthermore, based on SVMOP, we could also mine some predictive features such as useful polite expressions and sentiment words, which can be further applied to guide the development of mHealth platforms. Conclusions: It is the first time to model the problem of doctor performance evaluation as an ordinal regression problem. Experiments show that the performance of our proposed model with revised prediction function is better than many other machine learning methods on MAE, MSE as well as PAcc. In summary, with this model the mHealth platform could not only make an auto-evaluation of doctors’ performance online but also mine the most effective features, which could guide the promotion of doctors’ performance and the development of mHealth platforms.
- 18 July 2018
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 20 (7) , e240
Open AccessEvaluating Doctor Performance: Ordinal Regression-Based Approach
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Doctor performance evaluation (DPE) is an important task in mHealth, which aims to evaluate the overall quality of online diagnosis and patient outcomes so that high customer satisfaction and loyalty can be attained. However in reality most customers trend not to give ratings to doctors’ performance. Therefore it is imperative to develop a model to make DPE automatically. When making auto-evaluation of doctors’ performance, we expect to rate the doctors’ performance into a score label that is as close as possible to the true one. Objective: This study aims to do DPE automatically from online textual consultation contents between doctors and customers by a novel machine learning method. Methods: We propose a solution that models DPE as an ordinal regression problem. In doing so, a combined support vector machine and ordinal partitioning model (called SVMOP for short), along with an innovative prediction function is developed to capture the ordering labels hidden in preferences over DPE. Specifically, when conducting the feature engineering, in addition to the basic text features, eight handcrafted features extracted from over 70,000 medical entries, are added and then further boosted by Gradient Boosting Decision Tree. Results: Real data sets from one of the largest mobile doctor-patient communication platforms in China are used in our study. In according with the statistics, 64% of data on mHealth platforms lack the evaluation labels from customers. Experimental results reveal that our approach can well support doctor performance evaluation automatically. Specifically, compared with other auto-evaluation models, SVMOP improves mean absolute error (MAE) by 0.1, mean square error (MSE) by 0.5, pairwise accuracy (PAcc) by 5%; the handcrafted features that we suggest, improve MAE by 0.1, MSE by 0.2, PAcc by 3%. After boosting, the performance can be further improved. Furthermore, based on SVMOP, we could also mine some predictive features such as useful polite expressions and sentiment words, which can be further applied to guide the development of mHealth platforms. Conclusions: It is the first time to model the problem of doctor performance evaluation as an ordinal regression problem. Experiments show that the performance of our proposed model with revised prediction function is better than many other machine learning methods on MAE, MSE as well as PAcc. In summary, with this model the mHealth platform could not only make an auto-evaluation of doctors’ performance online but also mine the most effective features, which could guide the promotion of doctors’ performance and the development of mHealth platforms.