Sifei Lu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Sifei Lu has a total of 43 co-authors. Here are the top co-authors by number of shared publications:
Long Wang
Institute of High Performance Computing, Singapore
7 shared publicationsXiaorong Li
7 shared publicationsRick Siow Mong Goh
Institute of High Performance Computing (IHPC), Agency for Science Technology and Research (A∗STAR), Singapore
5 shared publicationsZengxiang Li
Institute of High Performance Computing (IHPC), Agency for Science Technology and Research (A∗STAR), Singapore
4 shared publicationsRubing Duan
Institute of High Performance Computing, Singapore
3 shared publicationsTerence Hung
3 shared publications
Latest Publications
- 1 January 2022
- Published by Springer Nature
- p. 69-81
An Interoperable Framework for Heterogeneous IoT infrastructure to Unlock Data Value
- 1 January 2022
- Published by Springer Nature
- p. 69-81
An Interoperable Framework for Heterogeneous IoT infrastructure to Unlock Data Value
- 1 December 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 160-169
Concurrent Hybrid Breadth-First-Search on Distributed PowerGraph for Skewed Graphs
Large-scale graph-structured computation is becoming increasingly important for various data analytics applications. However, most distributed graph processing frameworks do not directly support efficient implementation of sophisticated algorithms requiring massive graph traversals. In this paper, we propose concurrent hybrid breadth-first-search (BFS) algorithm on a popular distributed graph processing framework (Power-Graph and its optimized version PowerLyra). It leverages the small-world property of skewed graphs which apply to most realworld data sets. Hybrid BFS algorithm changes graph traversal direction to save computational workload and message transmissions. Concurrent BFS algorithm enables running multiple BFS simultaneously while sharing vertex explorations efficiently with bit operations. Extensive experiments are conducted to evaluate performance and scalability on a multi-core computer and a cluster of AWS instances. Concurrent hybrid BFS algorithm dramatically increases graph traversal efficiency with respect to the increasing number of concurrent BFS traversals. Compared with PowerGraph, PowerLyra uses fewer resources, while achieving significantly higher performance and better scalability.
- 1 December 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 160-169
Concurrent Hybrid Breadth-First-Search on Distributed PowerGraph for Skewed Graphs
Large-scale graph-structured computation is becoming increasingly important for various data analytics applications. However, most distributed graph processing frameworks do not directly support efficient implementation of sophisticated algorithms requiring massive graph traversals. In this paper, we propose concurrent hybrid breadth-first-search (BFS) algorithm on a popular distributed graph processing framework (Power-Graph and its optimized version PowerLyra). It leverages the small-world property of skewed graphs which apply to most realworld data sets. Hybrid BFS algorithm changes graph traversal direction to save computational workload and message transmissions. Concurrent BFS algorithm enables running multiple BFS simultaneously while sharing vertex explorations efficiently with bit operations. Extensive experiments are conducted to evaluate performance and scalability on a multi-core computer and a cluster of AWS instances. Concurrent hybrid BFS algorithm dramatically increases graph traversal efficiency with respect to the increasing number of concurrent BFS traversals. Compared with PowerGraph, PowerLyra uses fewer resources, while achieving significantly higher performance and better scalability.
- 1 December 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 319-323
A hybrid regression technique for house prices prediction
Usually, House price index represents the summarized price changes of residential housing. While for a single family house price prediction, it needs more accurate method based on location, house type, size, build year, local amenities, and some other factors which could affect house demand and supply. With limited dataset and data features, a practical and composite data pre-processing, creative feature engineering method is examined in this paper. The paper also proposes a hybrid Lasso and Gradient boosting regression model to predict individual house price. The proposed approach has recently been deployed as the key kernel for Kaggle Challenge “House Prices: Advanced Regression Techniques”. The performance is promising as our latest score was ranked top 1% out of all competition teams and individuals.
- 1 December 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 319-323
A hybrid regression technique for house prices prediction
Usually, House price index represents the summarized price changes of residential housing. While for a single family house price prediction, it needs more accurate method based on location, house type, size, build year, local amenities, and some other factors which could affect house demand and supply. With limited dataset and data features, a practical and composite data pre-processing, creative feature engineering method is examined in this paper. The paper also proposes a hybrid Lasso and Gradient boosting regression model to predict individual house price. The proposed approach has recently been deployed as the key kernel for Kaggle Challenge “House Prices: Advanced Regression Techniques”. The performance is promising as our latest score was ranked top 1% out of all competition teams and individuals.
- 20 October 2016
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9-16
Performance and Monetary Cost of Large-Scale Distributed Graph Processing on Amazon Cloud
Graph analytics has become essential to uncover relationship insights in complex systems. As graphs grow in scale, several graph-parallel frameworks including Pregel, GraphLab, and PowerGraph are developed based on commodity computers and/or Cloud instances. According to recent research and empirical performance evaluation, system optimization on PowerGraph allow it to outperform others significantly for processing natural graphs with skewed degree distribution. However, the performance characters, resource usage pattern and monetary cost have never been explored in-depth. In this paper, PowerGraph are evaluated with three different algorithms on Amazon EC2 instances with upto 768 CPU cores. We find that thegraph processing performance does not always increase with the increasing Cloud resources. Due to synchronization overheads, resources are not fully utilized. Graph processing tasks may prefer different execution strategies with specified number and type of Cloud instances to achieve high cost-efficiency, playing the trade-off between monetary cost and execution performance.
- 20 October 2016
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9-16
Performance and Monetary Cost of Large-Scale Distributed Graph Processing on Amazon Cloud
Graph analytics has become essential to uncover relationship insights in complex systems. As graphs grow in scale, several graph-parallel frameworks including Pregel, GraphLab, and PowerGraph are developed based on commodity computers and/or Cloud instances. According to recent research and empirical performance evaluation, system optimization on PowerGraph allow it to outperform others significantly for processing natural graphs with skewed degree distribution. However, the performance characters, resource usage pattern and monetary cost have never been explored in-depth. In this paper, PowerGraph are evaluated with three different algorithms on Amazon EC2 instances with upto 768 CPU cores. We find that thegraph processing performance does not always increase with the increasing Cloud resources. Due to synchronization overheads, resources are not fully utilized. Graph processing tasks may prefer different execution strategies with specified number and type of Cloud instances to achieve high cost-efficiency, playing the trade-off between monetary cost and execution performance.
- 1 June 2015
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 449-454
HiPerData: An autonomous large-scale model building and management platform for big data analytics
Data mining is a difficult task that relies on an exploratory and analytic process of processing large quantities of data in order to discover meaningful patterns for valuable insights. The increasing heterogeneity and complexity of data requires expert knowledge on how to combine multiple data mining techniques to process and analyze the data in an effective and efficient way. This paper presents a distributed architecture, HiPerData, for automated data processing and mining using large-scale computational resource management, model building and selection, and predictive and inference analysis. We illustrate two data mining tasks in which we automate the data mining knowledge flow construction based on the use of standards that have been defined in both data mining and automated-planning communities.
- 1 June 2015
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 449-454
HiPerData: An autonomous large-scale model building and management platform for big data analytics
Data mining is a difficult task that relies on an exploratory and analytic process of processing large quantities of data in order to discover meaningful patterns for valuable insights. The increasing heterogeneity and complexity of data requires expert knowledge on how to combine multiple data mining techniques to process and analyze the data in an effective and efficient way. This paper presents a distributed architecture, HiPerData, for automated data processing and mining using large-scale computational resource management, model building and selection, and predictive and inference analysis. We illustrate two data mining tasks in which we automate the data mining knowledge flow construction based on the use of standards that have been defined in both data mining and automated-planning communities.
- 1 December 2014
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 491-498
Hierarchical Parallelization and Runtime Scheduling for Pregel-Like Graph Processing Systems
Graph processing has become popular for various big data analytic applications. Google's Pregel framework enables vertex-centric graph processing in distributed environment based on Bulk Synchronous Parallel (BSP) model. However, the BSP model is inefficient for many complex graph algorithms requiring graph traversals, as only a small number of vertices really update states in each super step. In this paper, we propose an hierarchical parallelization mechanism, taking the advantages of both synchronous (warp-level) and asynchronous (task-level) parallelization approaches. In addition, a runtime task scheduling mechanism is proposed, relying on real-time monitoring or prediction of resource utilization. Experiments have verified that the hierarchical parallelization mechanism can expose greater parallelism, and thus, increase resource utilization significantly. Moreover, the runtime scheduling mechanism can avoid aggressive resource competition, and thus, further enhance the performance of the parallelized graph processing.
- 1 December 2014
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 491-498
Hierarchical Parallelization and Runtime Scheduling for Pregel-Like Graph Processing Systems
Graph processing has become popular for various big data analytic applications. Google's Pregel framework enables vertex-centric graph processing in distributed environment based on Bulk Synchronous Parallel (BSP) model. However, the BSP model is inefficient for many complex graph algorithms requiring graph traversals, as only a small number of vertices really update states in each super step. In this paper, we propose an hierarchical parallelization mechanism, taking the advantages of both synchronous (warp-level) and asynchronous (task-level) parallelization approaches. In addition, a runtime task scheduling mechanism is proposed, relying on real-time monitoring or prediction of resource utilization. Experiments have verified that the hierarchical parallelization mechanism can expose greater parallelism, and thus, increase resource utilization significantly. Moreover, the runtime scheduling mechanism can avoid aggressive resource competition, and thus, further enhance the performance of the parallelized graph processing.
- 1 December 2013
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 657-662
A Dynamic Hybrid Resource Provisioning Approach for Running Large-Scale Computational Applications on Cloud Spot and On-Demand Instances
Testing and executing large-scale computational applications in public clouds is becoming prevalent due to cost saving, elasticity, and scalability. However, how to increase the reliability and reduce the cost to run large-scale applications in public clouds is still a big challenge. In this paper, we analyzed the pricing schemes of Amazon Elastic Compute Cloud (EC2) and found the disturbance effect that the price of the spot instances can be heavily affected due to the large number of spot instances required. We proposed a dynamic approach which schedules and runs large-scale computational applications on a dynamic pool of cloud computational instances. We use hybrid instances, including both on-demand instances for high priority tasks and backup, and spot instances for normal computational tasks so as to further reduce the cost without significantly increasing the completion time. Our proposed method takes the dynamic pricing of cloud instances into consideration, and it reduces the cost and tolerates the failures for running large-scale applications in public clouds. We conducted experimental tests and an agent based Scalable complex System modeling for Sustainable city (S 3 ) application is used to evaluate the scalability, reliability and cost saving. The results show that our proposed method is robust and highly flexible for researchers and users to further reduce cost in real practice.
- 1 December 2013
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 657-662
A Dynamic Hybrid Resource Provisioning Approach for Running Large-Scale Computational Applications on Cloud Spot and On-Demand Instances
Testing and executing large-scale computational applications in public clouds is becoming prevalent due to cost saving, elasticity, and scalability. However, how to increase the reliability and reduce the cost to run large-scale applications in public clouds is still a big challenge. In this paper, we analyzed the pricing schemes of Amazon Elastic Compute Cloud (EC2) and found the disturbance effect that the price of the spot instances can be heavily affected due to the large number of spot instances required. We proposed a dynamic approach which schedules and runs large-scale computational applications on a dynamic pool of cloud computational instances. We use hybrid instances, including both on-demand instances for high priority tasks and backup, and spot instances for normal computational tasks so as to further reduce the cost without significantly increasing the completion time. Our proposed method takes the dynamic pricing of cloud instances into consideration, and it reduces the cost and tolerates the failures for running large-scale applications in public clouds. We conducted experimental tests and an agent based Scalable complex System modeling for Sustainable city (S 3 ) application is used to evaluate the scalability, reliability and cost saving. The results show that our proposed method is robust and highly flexible for researchers and users to further reduce cost in real practice.
- 1 August 2013
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 21618070,p. 207-212
RiskVis: Supply chain visualization with risk management and real-time monitoring
With increased complexity, supply chain networks (SCNs) of modern era face higher risks and lower efficiency due to limited visibility. Hence, there is an immediate need to provide end-to-end supply chain visibility for efficient management of complex supply chains. This paper proposes a visualization scheme based on multi-hierarchical modular design and develops a supply chain visualization platform with risk management and real-time monitoring, named RiskVis, for realizing better Supply Chain Risk Management (SCRM). A Supply Chain Visualizer (SCV) with a graphical visualization platform is mounted as a part of a SCRM management decision-making dashboard and it provides senior management a clearer view of supply chain operations in a local/regional/global setting. The platform not only displays spatio-temporal connectivity patterns of entities in a supply chain; it also accommodates real-time risk-related data collection and risk monitoring. The proposed platform offers the flexibility to be customized based on the user's requirements - to process and store the supply chain data in the server, visualize the supply chain data, network map, risk alert, and other information needed for SCRM. Supply chain decision makers can deploy it on the desktop or embed it into the company's enterprise applications in a front office environment for better managing risks of their supply chains.
- 1 August 2013
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 21618070,p. 207-212
RiskVis: Supply chain visualization with risk management and real-time monitoring
With increased complexity, supply chain networks (SCNs) of modern era face higher risks and lower efficiency due to limited visibility. Hence, there is an immediate need to provide end-to-end supply chain visibility for efficient management of complex supply chains. This paper proposes a visualization scheme based on multi-hierarchical modular design and develops a supply chain visualization platform with risk management and real-time monitoring, named RiskVis, for realizing better Supply Chain Risk Management (SCRM). A Supply Chain Visualizer (SCV) with a graphical visualization platform is mounted as a part of a SCRM management decision-making dashboard and it provides senior management a clearer view of supply chain operations in a local/regional/global setting. The platform not only displays spatio-temporal connectivity patterns of entities in a supply chain; it also accommodates real-time risk-related data collection and risk monitoring. The proposed platform offers the flexibility to be customized based on the user's requirements - to process and store the supply chain data in the server, visualize the supply chain data, network map, risk alert, and other information needed for SCRM. Supply chain decision makers can deploy it on the desktop or embed it into the company's enterprise applications in a front office environment for better managing risks of their supply chains.
- 1 July 2013
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 2324898X,p. 1049-1056
An Iterative Optimization Framework for Adaptive Workflow Management in Computational Clouds
As more and more data can be generated at a faster-than-ever rate nowadays, it becomes a challenge to processing large volumes of data for complex data analysis. In order to address performance and cost issues of big data processing on clouds, we present a novel design of adaptive workflow management system which includes a data mining based prediction model, workflow scheduler, and iteration controls to optimize the data processing via iterative workflow tasks. We proposed a new heuristic algorithm, called Upgrade Fit, which dynamically and continuously reallocates multiple types of cloud resources to fulfill the performance and cost requirements. The iterative workflow tasks can be bursty bags of tasks to be executed repetitively for data processing. A real application of weather forecast workflow has been used to evaluate the capability of our system for large volume image data processing. Experimental system has been set up and the results indicate that the system can effectively handle multiple types of cloud resources and optimize the performance iteratively.
- 1 July 2013
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 2324898X,p. 1049-1056
An Iterative Optimization Framework for Adaptive Workflow Management in Computational Clouds
As more and more data can be generated at a faster-than-ever rate nowadays, it becomes a challenge to processing large volumes of data for complex data analysis. In order to address performance and cost issues of big data processing on clouds, we present a novel design of adaptive workflow management system which includes a data mining based prediction model, workflow scheduler, and iteration controls to optimize the data processing via iterative workflow tasks. We proposed a new heuristic algorithm, called Upgrade Fit, which dynamically and continuously reallocates multiple types of cloud resources to fulfill the performance and cost requirements. The iterative workflow tasks can be bursty bags of tasks to be executed repetitively for data processing. A real application of weather forecast workflow has been used to evaluate the capability of our system for large volume image data processing. Experimental system has been set up and the results indicate that the system can effectively handle multiple types of cloud resources and optimize the performance iteratively.
- Open Access
- 18 June 2013
- Published by Association for Computing Machinery (ACM)
- p. 11-20
Evaluating hardware-assisted virtualization for deploying HPC-as-a-service
Virtualization has been the main driver behind the rise of Cloud computing. Despite Cloud computing's tremendous benefits to many applications (e.g., enterprise, Web, game/ multimedia, life sciences, and data analytics), its success in High Performance Computing (HPC) domain has been limited. The oft-cited reason is, apparently, latency caused by virtualization. Meanwhile, the rising popularity of virtualization has compelled CPU vendors to incorporate virtualization technology (VT) in chips. This hardware VT is believed to accelerate context switching, speed up memory address translation, and enable I/O direct access; those are basically sources of virtualization overheads. This paper reports the evaluation on computation and communication performance of different virtualized environments, i.e., Xen and KVM, leveraging hardware VT. Different network fabrics, namely Gigabit Ethernet and InfiniBand, were employed and tested in the virtualized environments and their results were compared against those in the native environments. A real-world HPC application (an MPI-based hydrodynamic simulation) was also used to assess the performance. Outcomes indicate that hardware-assisted virtualization can bring HPC-as-a-Service into realization.
- 18 June 2013
- Published by Association for Computing Machinery (ACM)
- p. 11-20
Open AccessEvaluating hardware-assisted virtualization for deploying HPC-as-a-service
Virtualization has been the main driver behind the rise of Cloud computing. Despite Cloud computing's tremendous benefits to many applications (e.g., enterprise, Web, game/ multimedia, life sciences, and data analytics), its success in High Performance Computing (HPC) domain has been limited. The oft-cited reason is, apparently, latency caused by virtualization. Meanwhile, the rising popularity of virtualization has compelled CPU vendors to incorporate virtualization technology (VT) in chips. This hardware VT is believed to accelerate context switching, speed up memory address translation, and enable I/O direct access; those are basically sources of virtualization overheads. This paper reports the evaluation on computation and communication performance of different virtualized environments, i.e., Xen and KVM, leveraging hardware VT. Different network fabrics, namely Gigabit Ethernet and InfiniBand, were employed and tested in the virtualized environments and their results were compared against those in the native environments. A real-world HPC application (an MPI-based hydrodynamic simulation) was also used to assess the performance. Outcomes indicate that hardware-assisted virtualization can bring HPC-as-a-Service into realization.