Sanghoon Kang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Sanghoon Kang has a total of 29 co-authors. Here are the top co-authors by number of shared publications:
- Hoi-Jun Yoo
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea
16 shared publications - Donghyeon Han
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology (MIT), Cambridge, MA, USA
13 shared publications - Dongseok Im
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea
7 shared publications - Sangyeob Kim
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea
7 shared publications - Jinsu Lee
School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea
6 shared publications - Gwangtae Park
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea
5 shared publications
Latest Publications
- 10 March 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Micro
- Vol. 43 (3) , 74-82
A Mobile 3-D Object Recognition Processor With Deep-Learning-Based Monocular Depth Estimation
A 3D object recognition system is a heavy task that consumes high sensor power and requires complex 3D data processing. In this article, the proposed processor produces 3D RGB-D data from an RGB image through a deep learning-based monocular depth estimation, and then its RGB-D data is sporadically calibrated with low-resolution depth data from a low-power depth sensor, lowering the sensor power by 27.3×. Then, the proposed processor accelerates various convolution operations in the system by integrating the in-out skipping-based bit-slice-level computing processing elements and flexibly allocating workloads considering data properties. Moreover, the point feature aggregator is designed close to the global memory to support the point feature reuse algorithm’s data aggregation. Additionally, the window-based search algorithm and its memory management are presented for efficient point processing in the point processing unit. Consequently, the 210 mW and 34 fps end-to-end 3D object recognition processor is successfully demonstrated.
- 10 March 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Micro
- Vol. 43 (3) , 74-82
A Mobile 3-D Object Recognition Processor With Deep-Learning-Based Monocular Depth Estimation
A 3D object recognition system is a heavy task that consumes high sensor power and requires complex 3D data processing. In this article, the proposed processor produces 3D RGB-D data from an RGB image through a deep learning-based monocular depth estimation, and then its RGB-D data is sporadically calibrated with low-resolution depth data from a low-power depth sensor, lowering the sensor power by 27.3×. Then, the proposed processor accelerates various convolution operations in the system by integrating the in-out skipping-based bit-slice-level computing processing elements and flexibly allocating workloads considering data properties. Moreover, the point feature aggregator is designed close to the global memory to support the point feature reuse algorithm’s data aggregation. Additionally, the window-based search algorithm and its memory management are presented for efficient point processing in the point processing unit. Consequently, the 210 mW and 34 fps end-to-end 3D object recognition processor is successfully demonstrated.
- 9 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 58 (1) , 177-188
DSPU: An Efficient Deep Learning-Based Dense RGB-D Data Acquisition With Sensor Fusion and 3-D Perception SoC
3-D red, green, blue, and depth (RGB-D) and 3-D perception are essential information for 3-D applications such as autonomous driving and augmented reality (AR)/virtual reality (VR) systems. However, battery-and resource-limited mobile devices face difficulties in obtaining dense RGB-D data and 3-D perception information in low-power (LP) and real-time. Specifically, an RGB-D sensor is used to acquire 3-D RGB-D data, but it consumes high power and produces sparse depth data. Moreover, preprocessing for RGB-D data requires a long execution time. Previous 3-D perception accelerators also have limited reconfigurability, making them incapable of executing diverse 3-D perception tasks. In this article, an LP and real-time depth signal processing system-on-chip (SoC), depth signal processing unit (DSPU), is presented. The DSPU produces accurate dense RGB-D data using a convolutional neural network (CNN)-based monocular depth estimation (MDE) and a sensor fusion with an LP ToF sensor. Then, the DSPU performs 3-D perception inferring a point cloud-based neural network (PNN). The DSPU executes the depth signal processing system with the following features: 1) a unified point processing unit (UPPU) with a flexible window based-search algorithm for simplifying the complexity of point processing algorithms and saving the arithmetic units and buffers; 2) a unified matrix processing unit (UMPU) with bit-slice-level sparsity exploitation to accelerate various matrix processing algorithms; 3) a band matrix encoder and decoder to decrease the data transactions in the conjugate-gradient (C-Grad) method; and 4) a point feature (PF) reuse method with a pipelined architecture for low-latency and LP PNN inference. Finally, the DSPU achieves real-time implementation with 281.6 mW of the end-to-end 3-D B-box extraction system.
- 9 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 58 (1) , 177-188
DSPU: An Efficient Deep Learning-Based Dense RGB-D Data Acquisition With Sensor Fusion and 3-D Perception SoC
3-D red, green, blue, and depth (RGB-D) and 3-D perception are essential information for 3-D applications such as autonomous driving and augmented reality (AR)/virtual reality (VR) systems. However, battery-and resource-limited mobile devices face difficulties in obtaining dense RGB-D data and 3-D perception information in low-power (LP) and real-time. Specifically, an RGB-D sensor is used to acquire 3-D RGB-D data, but it consumes high power and produces sparse depth data. Moreover, preprocessing for RGB-D data requires a long execution time. Previous 3-D perception accelerators also have limited reconfigurability, making them incapable of executing diverse 3-D perception tasks. In this article, an LP and real-time depth signal processing system-on-chip (SoC), depth signal processing unit (DSPU), is presented. The DSPU produces accurate dense RGB-D data using a convolutional neural network (CNN)-based monocular depth estimation (MDE) and a sensor fusion with an LP ToF sensor. Then, the DSPU performs 3-D perception inferring a point cloud-based neural network (PNN). The DSPU executes the depth signal processing system with the following features: 1) a unified point processing unit (UPPU) with a flexible window based-search algorithm for simplifying the complexity of point processing algorithms and saving the arithmetic units and buffers; 2) a unified matrix processing unit (UMPU) with bit-slice-level sparsity exploitation to accelerate various matrix processing algorithms; 3) a band matrix encoder and decoder to decrease the data transactions in the conjugate-gradient (C-Grad) method; and 4) a point feature (PF) reuse method with a pipelined architecture for low-latency and LP PNN inference. Finally, the DSPU achieves real-time implementation with 281.6 mW of the end-to-end 3-D B-box extraction system.
- Open Access
- 2 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of the Solid-State Circuits Society
- Vol. 2, 259-275
Energy-Efficient DNN Training Processors on Micro-AI Systems
Many edge/mobile devices are now able to utilize deep neural networks (DNNs) thanks to the development of mobile DNN accelerators. Mobile DNN accelerators overcame the problems of limited computing resources and battery capacity by realizing energy-efficient inference. However, its passive behavior makes it difficult for DNN to provide active customization for individual users or its service environment. The importance of on-chip training is rising more and more to provide active interaction between DNN processors and ever-changing surroundings or conditions. Despite its advantages, the DNN training has more constraints than the inference such that it was considered impractical to be realized on mobile/edge devices. Recently, there are many trials to realize mobile DNN training, and a number of prior works will be summarized. Firstly, it arranges the new challenges of the DNN accelerator induced by training functionality and discusses new hardware features related to the challenges. Secondly, it explains algorithm-hardware co-optimization methods and explains why it becomes mainstream in mobile DNN training research. Thirdly, it compares the main differences between the conventional inference accelerators and recent training processors. Finally, the conclusion is made by proposing the future directions of the DNN training processor in micro-AI systems.
- 2 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of the Solid-State Circuits Society
- Vol. 2, 259-275
Open AccessEnergy-Efficient DNN Training Processors on Micro-AI Systems
Many edge/mobile devices are now able to utilize deep neural networks (DNNs) thanks to the development of mobile DNN accelerators. Mobile DNN accelerators overcame the problems of limited computing resources and battery capacity by realizing energy-efficient inference. However, its passive behavior makes it difficult for DNN to provide active customization for individual users or its service environment. The importance of on-chip training is rising more and more to provide active interaction between DNN processors and ever-changing surroundings or conditions. Despite its advantages, the DNN training has more constraints than the inference such that it was considered impractical to be realized on mobile/edge devices. Recently, there are many trials to realize mobile DNN training, and a number of prior works will be summarized. Firstly, it arranges the new challenges of the DNN accelerator induced by training functionality and discusses new hardware features related to the challenges. Secondly, it explains algorithm-hardware co-optimization methods and explains why it becomes mainstream in mobile DNN training research. Thirdly, it compares the main differences between the conventional inference accelerators and recent training processors. Finally, the conclusion is made by proposing the future directions of the DNN training processor in micro-AI systems.
- 21 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-25
DSPU: A 281.6mW Real-Time Deep Learning-Based Dense RGB-D Data Acquisition with Sensor Fusion and 3D Perception System-on-Chip
3D Data in Mobile Platforms
- 21 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-25
DSPU: A 281.6mW Real-Time Deep Learning-Based Dense RGB-D Data Acquisition with Sensor Fusion and 3D Perception System-on-Chip
3D Data in Mobile Platforms
- 1 August 2022
- Published by Academy of Management in Academy of Management Proceedings
- Vol. 2022 (1)
Beating the Rival but Losing the Game: How Rivalry Shifts Negotiator Goals Away from Value Claiming
Decades of negotiations research has revealed the importance of BATNA – best alternatives to negotiated agreement. Negotiators with stronger BATNAs tend to have greater power and claim higher value...
- 1 August 2022
- Published by Academy of Management in Academy of Management Proceedings
- Vol. 2022 (1)
Beating the Rival but Losing the Game: How Rivalry Shifts Negotiator Goals Away from Value Claiming
Decades of negotiations research has revealed the importance of BATNA – best alternatives to negotiated agreement. Negotiators with stronger BATNAs tend to have greater power and claim higher value...
- 4 July 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 57 (10) , 2944-2956
FlashMAC: A Time-Frequency Hybrid MAC Architecture With Variable Latency-Aware Scheduling for TinyML Systems
With the widespread of deep neural networks (DNNs) in diverse applications, tiny platforms such as Internet-of-Things devices are starting to adopt DNNs. Due to their extreme energy and form factor constraints, conventional digital-only implementations of multiply-and-accumulate (MAC) acceleration faced fundamental limitations. To that end, the investigation into mixed-signal computing architectures is growing rapidly. Motivated by the flash ADC, this article proposes FlashMAC architecture that can natively support multibit multiplication. In addition, through fusing time- and frequency-domain computing methods without power-hungry oscillators, it enables low latency accumulation with low power consumption. As a result, the proposed time-frequency hybrid architecture achieves high energy efficiency with the support for complex DNN models requiring higher precision. To enhance the robustness of PVT variation of the mixed-signal architecture, a frequency calibration loop is integrated. In addition, motivated by the data-dependent performance of the FlashMAC architecture, variable latency-aware scheduling is proposed. The FlashMAC does not skip MAC operations as zero-skipping architectures do, but the latency of the operation can be lower when operands are smaller in magnitude. Tackling the issue through software and hardware co-optimization, loose synchronization architecture and magnitude-aware weight reordering increase the DNN benchmark performance by achieving higher utilization of the parallel FlashMAC array. The proposed features are integrated into a test chip which is fabricated in 65-nm logic CMOS technology. The silicon chip achieves 56.52 TOPS/W peak energy efficiency and a peak operating frequency of 90 MHz. Tested with the VGG16 benchmark trained on the Imagenet dataset, it achieved 17.04-ms latency while showing 11.15 TOPS/W energy efficiency. As a result, compared to the previous state-of-the-art, the proposed FlashMAC achieved 3.15 $\times $ higher normalized energy efficiency.
- 4 July 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 57 (10) , 2944-2956
FlashMAC: A Time-Frequency Hybrid MAC Architecture With Variable Latency-Aware Scheduling for TinyML Systems
With the widespread of deep neural networks (DNNs) in diverse applications, tiny platforms such as Internet-of-Things devices are starting to adopt DNNs. Due to their extreme energy and form factor constraints, conventional digital-only implementations of multiply-and-accumulate (MAC) acceleration faced fundamental limitations. To that end, the investigation into mixed-signal computing architectures is growing rapidly. Motivated by the flash ADC, this article proposes FlashMAC architecture that can natively support multibit multiplication. In addition, through fusing time- and frequency-domain computing methods without power-hungry oscillators, it enables low latency accumulation with low power consumption. As a result, the proposed time-frequency hybrid architecture achieves high energy efficiency with the support for complex DNN models requiring higher precision. To enhance the robustness of PVT variation of the mixed-signal architecture, a frequency calibration loop is integrated. In addition, motivated by the data-dependent performance of the FlashMAC architecture, variable latency-aware scheduling is proposed. The FlashMAC does not skip MAC operations as zero-skipping architectures do, but the latency of the operation can be lower when operands are smaller in magnitude. Tackling the issue through software and hardware co-optimization, loose synchronization architecture and magnitude-aware weight reordering increase the DNN benchmark performance by achieving higher utilization of the parallel FlashMAC array. The proposed features are integrated into a test chip which is fabricated in 65-nm logic CMOS technology. The silicon chip achieves 56.52 TOPS/W peak energy efficiency and a peak operating frequency of 90 MHz. Tested with the VGG16 benchmark trained on the Imagenet dataset, it achieved 17.04-ms latency while showing 11.15 TOPS/W energy efficiency. As a result, compared to the previous state-of-the-art, the proposed FlashMAC achieved 3.15 $\times $ higher normalized energy efficiency.
- 20 April 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-3
A Low-power and Real-time 3D Object Recognition Processor with Dense RGB-D Data Acquisition in Mobile Platforms
A low-power and real-time 3D object recognition with RGBD data acquisition system-on-chip (SoC) is proposed. By synthesizing dense RGB-D data through monocular depth estimation, the proposed system reduces the sensor power for 3D data acquisition by ×27.3 lower. Moreover, the proposed processor reduces the energy consumption of a point cloud based neural network (PNN) exploiting bit-slice-level computation and a point feature reuse method with a pipelined architecture. Additionally, the processor supports the point sampling and grouping algorithms of the PNN with a unified point processing core. Finally, the processor achieves 210.0 mW while implementing 34.0 frame-per-second (fps) end-to-end RGB-D acquisition and 3D object recognition.
- 20 April 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-3
A Low-power and Real-time 3D Object Recognition Processor with Dense RGB-D Data Acquisition in Mobile Platforms
A low-power and real-time 3D object recognition with RGBD data acquisition system-on-chip (SoC) is proposed. By synthesizing dense RGB-D data through monocular depth estimation, the proposed system reduces the sensor power for 3D data acquisition by ×27.3 lower. Moreover, the proposed processor reduces the energy consumption of a point cloud based neural network (PNN) exploiting bit-slice-level computation and a point feature reuse method with a pipelined architecture. Additionally, the processor supports the point sampling and grouping algorithms of the PNN with a unified point processing core. Finally, the processor achieves 210.0 mW while implementing 34.0 frame-per-second (fps) end-to-end RGB-D acquisition and 3D object recognition.
- 4 January 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems I: Regular Papers
- Vol. 69 (4) , 1494-1506
TSUNAMI: Triple Sparsity-Aware Ultra Energy-Efficient Neural Network Training Accelerator With Multi-Modal Iterative Pruning
This article proposes the TSUNAMI, which supports an energy-efficient deep-neural-network training. The TSUNAMI supports multi-modal iterative pruning to generate zeros in activation and weight. Tile-based dynamic activation pruning unit and weight memory shared pruning unit eliminate additional memory access. Coarse-zero skipping controller skips multiple unnecessary multiply-and-accumulation (MAC) operations at once, and fine-zero skipping controller skips randomly located unnecessary MAC operations. Weight sparsity balancer solves a utilization degradation caused by weight sparsity imbalance, and the workload of each convolution core is allocated by a random channel allocator. The TSUNAMI achieves an energy efficiency of 3.42 TFLOPS/W at 0.78V and 50MHz with floating-point 8-bit activation and weight. Also, it achieves an energy efficiency of 405.96 TFLOPS/W at 90% sparsity condition.
- 4 January 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems I: Regular Papers
- Vol. 69 (4) , 1494-1506
TSUNAMI: Triple Sparsity-Aware Ultra Energy-Efficient Neural Network Training Accelerator With Multi-Modal Iterative Pruning
This article proposes the TSUNAMI, which supports an energy-efficient deep-neural-network training. The TSUNAMI supports multi-modal iterative pruning to generate zeros in activation and weight. Tile-based dynamic activation pruning unit and weight memory shared pruning unit eliminate additional memory access. Coarse-zero skipping controller skips multiple unnecessary multiply-and-accumulation (MAC) operations at once, and fine-zero skipping controller skips randomly located unnecessary MAC operations. Weight sparsity balancer solves a utilization degradation caused by weight sparsity imbalance, and the workload of each convolution core is allocated by a random channel allocator. The TSUNAMI achieves an energy efficiency of 3.42 TFLOPS/W at 0.78V and 50MHz with floating-point 8-bit activation and weight. Also, it achieves an energy efficiency of 405.96 TFLOPS/W at 90% sparsity condition.
- 15 October 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal on Emerging and Selected Topics in Circuits and Systems
- Vol. 11 (4) , 634-648
An Overview of Sparsity Exploitation in CNNs for On-Device Intelligence With Software-Hardware Cross-Layer Optimizations
This paper presents a detailed overview of sparsity exploitation in deep neural network (DNN) accelerators. Despite the algorithmic advancements which drove DNNs to become the standard of artificial intelligence (AI), DNNs’ computational and memory overhead limits the deployment of off-the-shelf models on edge devices. Numerous optimizations have been widely studied to efficiently run DNN models on performance and energy-limited mobile devices, from both software’s and hardware’s perspective. Sparsity exploitation one of the mainstream optimization techniques, whose objective is to achieve higher efficiency and speed through avoiding redundant multiply-and-accumulate (MAC) operations cause by zero operands. This paper overviews previous contributions on sparsity exploitation from both the software and hardware side of views, with the newly proposed taxonomy to categorize and analyze the works. On the software side, different sparsification algorithms are explained, including pruning and output speculation. From the hardware’s perspective, advancements in architectures to efficiently handle sparse DNN computation are elaborated. The proposed taxonomy will help classify previous accelerators easily, by which sparsity they exploit and how. In addition, related works on sparse processing-in-memory (PIM) architectures and similarity exploitation are briefly introduced.
- 15 October 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal on Emerging and Selected Topics in Circuits and Systems
- Vol. 11 (4) , 634-648
An Overview of Sparsity Exploitation in CNNs for On-Device Intelligence With Software-Hardware Cross-Layer Optimizations
This paper presents a detailed overview of sparsity exploitation in deep neural network (DNN) accelerators. Despite the algorithmic advancements which drove DNNs to become the standard of artificial intelligence (AI), DNNs’ computational and memory overhead limits the deployment of off-the-shelf models on edge devices. Numerous optimizations have been widely studied to efficiently run DNN models on performance and energy-limited mobile devices, from both software’s and hardware’s perspective. Sparsity exploitation one of the mainstream optimization techniques, whose objective is to achieve higher efficiency and speed through avoiding redundant multiply-and-accumulate (MAC) operations cause by zero operands. This paper overviews previous contributions on sparsity exploitation from both the software and hardware side of views, with the newly proposed taxonomy to categorize and analyze the works. On the software side, different sparsification algorithms are explained, including pruning and output speculation. From the hardware’s perspective, advancements in architectures to efficiently handle sparse DNN computation are elaborated. The proposed taxonomy will help classify previous accelerators easily, by which sparsity they exploit and how. In addition, related works on sparse processing-in-memory (PIM) architectures and similarity exploitation are briefly introduced.
- 15 July 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 56 (10) , 2968-2980
An Energy-Efficient GAN Accelerator With On-Chip Training for Domain-Specific Optimization
Generative adversarial networks (GANs) consist of multiple deep neural networks cooperating and competing with each other. Due to their complex architectures and large feature map sizes, training GANs requires a huge amount of computations. Moreover, instance normalization (IN) layers in GANs dramatically increase the external memory access (EMA). However, retraining GANs with user-specific data is critical on mobile devices because the pre-trained model outputs distorted images under user-specific conditions. This article proposes a GAN training accelerator to enable energy-efficient domain-specific optimization of GAN with user’s local data. Selective layer retraining (SELRET) picks out layers that are effective in enhancing the quality of the retrained model. Without image quality degradation, the SELRET reduces the required computation by 69%. Moreover, reordering layers for instance normalization (ROLIN) is proposed to reduce the EMA of intermediate data. Through the implementation of the proposed architecture, which splits and reorders the IN layers, 38.7% and 32.2% of overall EMA reduction are achieved in the forward propagation (FP) stage and the error propagation (EP) stage, respectively. The proposed processor is fabricated in a 65-nm CMOS process, showing 0.38-TFLOPS/W energy efficiency. The chip can retrain a face modification GAN with a custom dataset of 256$\times $ 256 images over 100 epochs under 30 s while only consuming 274 mW. Compared to the previous FPGA implementation, this work improved the retraining performance and energy efficiency by 2$\times $ and 39$\times $ , respectively. As a result, the proposed accelerator enables GAN’s domain-specific optimization on a mobile platform.- 15 July 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 56 (10) , 2968-2980
An Energy-Efficient GAN Accelerator With On-Chip Training for Domain-Specific Optimization
Generative adversarial networks (GANs) consist of multiple deep neural networks cooperating and competing with each other. Due to their complex architectures and large feature map sizes, training GANs requires a huge amount of computations. Moreover, instance normalization (IN) layers in GANs dramatically increase the external memory access (EMA). However, retraining GANs with user-specific data is critical on mobile devices because the pre-trained model outputs distorted images under user-specific conditions. This article proposes a GAN training accelerator to enable energy-efficient domain-specific optimization of GAN with user’s local data. Selective layer retraining (SELRET) picks out layers that are effective in enhancing the quality of the retrained model. Without image quality degradation, the SELRET reduces the required computation by 69%. Moreover, reordering layers for instance normalization (ROLIN) is proposed to reduce the EMA of intermediate data. Through the implementation of the proposed architecture, which splits and reorders the IN layers, 38.7% and 32.2% of overall EMA reduction are achieved in the forward propagation (FP) stage and the error propagation (EP) stage, respectively. The proposed processor is fabricated in a 65-nm CMOS process, showing 0.38-TFLOPS/W energy efficiency. The chip can retrain a face modification GAN with a custom dataset of 256$\times $ 256 images over 100 epochs under 30 s while only consuming 274 mW. Compared to the previous FPGA implementation, this work improved the retraining performance and energy efficiency by 2$\times $ and 39$\times $ , respectively. As a result, the proposed accelerator enables GAN’s domain-specific optimization on a mobile platform.