Arash Fayyazi
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Arash Fayyazi has a total of 54 co-authors. Here are the top co-authors by number of shared publications:
- Massoud Pedram
University of Southern California, USA
17 shared publications - Amirhossein Esmaili
Department of Electrical Engineering University of Southern California Los Angeles California
4 shared publications - Mahdi Nazemi
Department of Electrical & Computer Engineering University of Southern California Los Angeles CA USA
4 shared publications - Mehdi Kamal
University of Southern California, USA
4 shared publications - Shahin Nazarian
University of Southern California Los Angeles CA
4 shared publications - Ali Afzali-Kusha
University of Tehran
3 shared publications
Latest Publications
- Open Access
- 24 January 2025
- Published by IOP Publishing in Superconductor Science and Technology
- Vol. 38  (2) , 025014
Scalable superconductor neuron with ternary synaptic connections for ultra-fast SNN hardware
A novel high-fan-in differential superconductor neuron structure designed for ultra-high-performance spiking neural network (SNN) accelerators is presented. Utilizing a high-fan-in neuron structure allows us to design SNN accelerators with more synaptic connections, enhancing the overall network capabilities. The proposed neuron design is based on superconductor electronics fabric, incorporating multiple superconducting loops, each with two Josephson Junctions. This arrangement enables each input data branch to have positive and negative inductive coupling, supporting excitatory and inhibitory synaptic data. Compatibility with synaptic devices and thresholding operation is achieved using a single flux quantum pulse-based logic style. The neuron design, along with ternary synaptic connections, forms the foundation for a superconductor-based SNN inference. To demonstrate the capabilities of our design, we train the SNN using snnTorch, augmenting the PyTorch framework. After pruning, the demonstrated SNN inference achieves an impressive 96.1% accuracy on MNIST images. Notably, the network exhibits a remarkable throughput of 8.92 GHz while consuming only 1.5 nJ per inference, including the energy consumption associated with cooling to 4 K. These results underscore the potential of superconductor electronics in developing high-performance and ultra-energy-efficient neural network accelerator architectures.
- 24 January 2025
- Published by IOP Publishing in Superconductor Science and Technology
- Vol. 38  (2) , 025014
Open AccessScalable superconductor neuron with ternary synaptic connections for ultra-fast SNN hardware
A novel high-fan-in differential superconductor neuron structure designed for ultra-high-performance spiking neural network (SNN) accelerators is presented. Utilizing a high-fan-in neuron structure allows us to design SNN accelerators with more synaptic connections, enhancing the overall network capabilities. The proposed neuron design is based on superconductor electronics fabric, incorporating multiple superconducting loops, each with two Josephson Junctions. This arrangement enables each input data branch to have positive and negative inductive coupling, supporting excitatory and inhibitory synaptic data. Compatibility with synaptic devices and thresholding operation is achieved using a single flux quantum pulse-based logic style. The neuron design, along with ternary synaptic connections, forms the foundation for a superconductor-based SNN inference. To demonstrate the capabilities of our design, we train the SNN using snnTorch, augmenting the PyTorch framework. After pruning, the demonstrated SNN inference achieves an impressive 96.1% accuracy on MNIST images. Notably, the network exhibits a remarkable throughput of 8.92 GHz while consuming only 1.5 nJ per inference, including the energy consumption associated with cooling to 4 K. These results underscore the potential of superconductor electronics in developing high-performance and ultra-energy-efficient neural network accelerator architectures.
- Open Access
- 5 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-6
PEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers
The deployment of Vision Transformers (ViTs) on hardware platforms, specially Field-Programmable Gate Arrays (FPGAs), presents many challenges, which are mainly due to the substantial computational and power requirements of their non-linear functions, notably layer normalization, softmax, and Gaussian Error Linear Unit (GELU). These critical functions pose significant obstacles to efficient hardware implementation due to their complex mathematical operations and the inherent resource count and architectural limitations of FPGAs. PEANO-ViT offers a novel approach to streamlining the implementation of the layer normalization layer by introducing a division-free technique that simultaneously approximates the division and square root function. Additionally, PEANO-ViT provides a multi-scale division strategy to eliminate division operations in the softmax layer, aided by a Padé-based approximation for the exponential function. Finally, PEANO-ViT introduces a piece-wise linear approximation for the GELU function, carefully designed to bypass the computationally intensive operations associated with GELU. In our comprehensive evaluations, PEANO-ViT exhibits minimal accuracy degradation ≤ 0.5% for DeiT-B) while significantly enhancing power efficiency, achieving improvements of 1.91X, 1.39X, and 8.01X for layer normalization, softmax, and GELU, respectively. This improvement is achieved through substantial reductions in DSP, LUT, and register counts for these non-linear operations. Consequently, PEANO-ViT enables efficient deployment of Vision Transformers on resource- and power-constrained FPGA platforms.
- 5 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-6
Open AccessPEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers
The deployment of Vision Transformers (ViTs) on hardware platforms, specially Field-Programmable Gate Arrays (FPGAs), presents many challenges, which are mainly due to the substantial computational and power requirements of their non-linear functions, notably layer normalization, softmax, and Gaussian Error Linear Unit (GELU). These critical functions pose significant obstacles to efficient hardware implementation due to their complex mathematical operations and the inherent resource count and architectural limitations of FPGAs. PEANO-ViT offers a novel approach to streamlining the implementation of the layer normalization layer by introducing a division-free technique that simultaneously approximates the division and square root function. Additionally, PEANO-ViT provides a multi-scale division strategy to eliminate division operations in the softmax layer, aided by a Padé-based approximation for the exponential function. Finally, PEANO-ViT introduces a piece-wise linear approximation for the GELU function, carefully designed to bypass the computationally intensive operations associated with GELU. In our comprehensive evaluations, PEANO-ViT exhibits minimal accuracy degradation ≤ 0.5% for DeiT-B) while significantly enhancing power efficiency, achieving improvements of 1.91X, 1.39X, and 8.01X for layer normalization, softmax, and GELU, respectively. This improvement is achieved through substantial reductions in DSP, LUT, and register counts for these non-linear operations. Consequently, PEANO-ViT enables efficient deployment of Vision Transformers on resource- and power-constrained FPGA platforms.
- Open Access
- 12 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 730-735
NeuroBlend: Towards Low-Power yet Accurate Neural Network-Based Inference Engine Blending Binary and Fixed-Point Convolutions
- 12 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 730-735
Open AccessNeuroBlend: Towards Low-Power yet Accurate Neural Network-Based Inference Engine Blending Binary and Fixed-Point Convolutions
- 18 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Applied Superconductivity
- Vol. 33  (8) , 1-26
Results From the ColdFlux Superconductor Integrated Circuit Design Tool Project
In five and a half years, the ColdFlux project under the IARPA SuperTools program pushed the boundaries of digital and analog superconductor electronic design automation (S-EDA) tools. The SuperTools program demanded significant beyond-state-of-the-art deliverables in four main areas: RTL synthesis, architectures, and verification; analog design and layout synthesis; physical design and test; and technology CAD and cell library design. Through the work of academic groups scattered over four continents, the ColdFlux effort forged into a powerful set of open-source and commercial S-EDA tools unlike any before, rivaled only by a commercial toolchain from Synopsys under the same SuperTools umbrella. We present an overview of the tools from where we started to the eventual project deliverables. These include powerful simulation and extraction engines, magnetic field and flux trapping analysis, advanced clocking methods, multi-chip interface extraction and verification, unified multi-layer design-rule compliant track blocks for automated place and route of both rapid single flux quantum (RSFQ) and adiabatic quantum-flux-parametron (AQFP) cells, models and tools for validation and test, multi-bit single flux quantum (SFQ) cells, architecture innovations for full CPU designs and more. Comprehensive cell libraries and a process design kit (PDK) were developed with the ColdFlux tools. The AQFP cell library features a logically rich collection of 80+ cells, including 3- and 5-input logic gates, signal-driving boosters, and refined RSFQ-to-AQFP interfaces, while the RSFQ library has 30+ cells. Finally, we discuss how the full toolchain enables and enhances the superconductor IC design process.
- 18 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Applied Superconductivity
- Vol. 33  (8) , 1-26
Results From the ColdFlux Superconductor Integrated Circuit Design Tool Project
In five and a half years, the ColdFlux project under the IARPA SuperTools program pushed the boundaries of digital and analog superconductor electronic design automation (S-EDA) tools. The SuperTools program demanded significant beyond-state-of-the-art deliverables in four main areas: RTL synthesis, architectures, and verification; analog design and layout synthesis; physical design and test; and technology CAD and cell library design. Through the work of academic groups scattered over four continents, the ColdFlux effort forged into a powerful set of open-source and commercial S-EDA tools unlike any before, rivaled only by a commercial toolchain from Synopsys under the same SuperTools umbrella. We present an overview of the tools from where we started to the eventual project deliverables. These include powerful simulation and extraction engines, magnetic field and flux trapping analysis, advanced clocking methods, multi-chip interface extraction and verification, unified multi-layer design-rule compliant track blocks for automated place and route of both rapid single flux quantum (RSFQ) and adiabatic quantum-flux-parametron (AQFP) cells, models and tools for validation and test, multi-bit single flux quantum (SFQ) cells, architecture innovations for full CPU designs and more. Comprehensive cell libraries and a process design kit (PDK) were developed with the ColdFlux tools. The AQFP cell library features a logically rich collection of 80+ cells, including 3- and 5-input logic gates, signal-driving boosters, and refined RSFQ-to-AQFP interfaces, while the RSFQ library has 30+ cells. Finally, we discuss how the full toolchain enables and enhances the superconductor IC design process.
- Open Access
CrAFT: Compression-Aware Fine-Tuning for Efficient Visual Task Adaptation
Transfer learning has become a popular task adaptation method in the era of foundation models. However, many foundation models require large storage and computing resources, which makes off-the-shelf deployment impractical. Post-training compression techniques such as pruning and quantization can help lower deployment costs. Unfortunately, the resulting performance degradation limits the usability and benefits of such techniques. To close this performance gap, we propose CrAFT, a simple fine-tuning framework that enables effective post-training network compression. In CrAFT, users simply employ the default fine-tuning schedule along with sharpness minimization objective, simultaneously facilitating task adaptation and compression-friendliness. Contrary to the conventional sharpness minimization techniques, which are applied during pretraining, the CrAFT approach adds negligible training overhead as fine-tuning is done in under a couple of minutes or hours with a single GPU. The effectiveness of CrAFT, which is a general-purpose tool that can significantly boost one-shot pruning and post-training quantization, is demonstrated on both convolution-based and attention-based vision foundation models on a variety of target tasks. The code will be made publicly available.
Open AccessCrAFT: Compression-Aware Fine-Tuning for Efficient Visual Task Adaptation
Transfer learning has become a popular task adaptation method in the era of foundation models. However, many foundation models require large storage and computing resources, which makes off-the-shelf deployment impractical. Post-training compression techniques such as pruning and quantization can help lower deployment costs. Unfortunately, the resulting performance degradation limits the usability and benefits of such techniques. To close this performance gap, we propose CrAFT, a simple fine-tuning framework that enables effective post-training network compression. In CrAFT, users simply employ the default fine-tuning schedule along with sharpness minimization objective, simultaneously facilitating task adaptation and compression-friendliness. Contrary to the conventional sharpness minimization techniques, which are applied during pretraining, the CrAFT approach adds negligible training overhead as fine-tuning is done in under a couple of minutes or hours with a single GPU. The effectiveness of CrAFT, which is a general-purpose tool that can significantly boost one-shot pruning and post-training quantization, is demonstrated on both convolution-based and attention-based vision foundation models on a variety of target tasks. The code will be made publicly available.
- Open Access
Algorithms and Hardware for Efficient Processing of Logic-based Neural Networks
Recent efforts to improve the performance of neural network (NN) accelerators that meet today's application requirements have given rise to a new trend of logic-based NN inference relying on fixed-function combinational logic (FFCL). This paper presents an innovative optimization methodology for compiling and mapping NNs utilizing FFCL into a logic processor. The presented method maps FFCL blocks to a set of Boolean functions where Boolean operations in each function are mapped to high-performance, low-latency, parallelized processing elements. Graph partitioning and scheduling algorithms are presented to handle FFCL blocks that cannot straightforwardly fit the logic processor. Our experimental evaluations across several datasets and NNs demonstrate the superior performance of our framework in terms of the inference throughput compared to prior art NN accelerators. We achieve 25x higher throughput compared with the XNOR-based accelerator for VGG16 model that can be amplified 5x deploying the graph partitioning and merging algorithms.
Open AccessAlgorithms and Hardware for Efficient Processing of Logic-based Neural Networks
Recent efforts to improve the performance of neural network (NN) accelerators that meet today's application requirements have given rise to a new trend of logic-based NN inference relying on fixed-function combinational logic (FFCL). This paper presents an innovative optimization methodology for compiling and mapping NNs utilizing FFCL into a logic processor. The presented method maps FFCL blocks to a set of Boolean functions where Boolean operations in each function are mapped to high-performance, low-latency, parallelized processing elements. Graph partitioning and scheduling algorithms are presented to handle FFCL blocks that cannot straightforwardly fit the logic processor. Our experimental evaluations across several datasets and NNs demonstrate the superior performance of our framework in terms of the inference throughput compared to prior art NN accelerators. We achieve 25x higher throughput compared with the XNOR-based accelerator for VGG16 model that can be amplified 5x deploying the graph partitioning and merging algorithms.
- 7 April 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Applied Superconductivity
- Vol. 33  (5) , 1-5
Formal Verification of Sequential Circuits in Superconducting Single Flux Quantum Technologies
Logical equivalence checking is one of the most effective verification steps in the entire design process of integrated circuits. However, it faces challenges in emerging technologies due to differences between its logic models and those of the standard complementary metal oxide semiconductor (CMOS). This paper presents qSC, an equivalence checking framework targeting sequential circuits mapped to rapid superconducting single-flux-quantum (RSFQ) logic circuits. In addition to a logical checking module, qSC also includes several structural checking modules. The structural checking modules are used to check whether the circuit meets the design rules of superconducting RSFQ logic circuits. The main difficulty of verifying nonlinear sequential circuits is in finding the levels of logic cells in the circuit for full-path-balancing checking for which the conventional topological sorting algorithm of acyclic digraphs is not applicable. The key is to define and calculate levels for nodes in a cyclic graph and to avoid any loop trap. Hence, a new algorithm is presented for full-path-balancing checking of the cyclic digraph that satisfies all of these desirable features. The experimental results show the feasibility of qSC on sequential circuits implemented in RSFQ technologies.
- 7 April 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Applied Superconductivity
- Vol. 33  (5) , 1-5
Formal Verification of Sequential Circuits in Superconducting Single Flux Quantum Technologies
Logical equivalence checking is one of the most effective verification steps in the entire design process of integrated circuits. However, it faces challenges in emerging technologies due to differences between its logic models and those of the standard complementary metal oxide semiconductor (CMOS). This paper presents qSC, an equivalence checking framework targeting sequential circuits mapped to rapid superconducting single-flux-quantum (RSFQ) logic circuits. In addition to a logical checking module, qSC also includes several structural checking modules. The structural checking modules are used to check whether the circuit meets the design rules of superconducting RSFQ logic circuits. The main difficulty of verifying nonlinear sequential circuits is in finding the levels of logic cells in the circuit for full-path-balancing checking for which the conventional topological sorting algorithm of acyclic digraphs is not applicable. The key is to define and calculate levels for nodes in a cyclic graph and to avoid any loop trap. Hence, a new algorithm is presented for full-path-balancing checking of the cyclic digraph that satisfies all of these desirable features. The experimental results show the feasibility of qSC on sequential circuits implemented in RSFQ technologies.
- Open Access
- 11 March 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Reconfigurable Technology and Systems
- Vol. 16  (2) , 1-25
Efficient Compilation and Mapping of Fixed Function Combinational Logic onto Digital Signal Processors Targeting Neural Network Inference and Utilizing High-level Synthesis
Recent efforts for improving the performance of neural network (NN) accelerators that meet today’s application requirements have given rise to a new trend of logic-based NN inference relying on fixed function combinational logic. Mapping such large Boolean functions with many input variables and product terms to digital signal processors (DSPs) on Field-programmable gate arrays (FPGAs) needs a novel framework considering the structure and reconfigurability of DSP blocks during this process. The proposed methodology in this article maps the fixed function combinational logic blocks to a set of Boolean functions where Boolean operations corresponding to each function are mapped to DSP devices rather than look-up tables on the FPGAs to take advantage of the high performance, low latency, and parallelism of DSP blocks. This article also presents an innovative design and optimization methodology for compilation and mapping of NNs, utilizing fixed function combinational logic to DSPs on FPGAs employing high-level synthesis flow. Our experimental evaluations across several datasets and selected NNs demonstrate the comparable performance of our framework in terms of the inference latency and output accuracy compared to prior art FPGA-based NN accelerators employing DSPs.
- 11 March 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Reconfigurable Technology and Systems
- Vol. 16  (2) , 1-25
Open AccessEfficient Compilation and Mapping of Fixed Function Combinational Logic onto Digital Signal Processors Targeting Neural Network Inference and Utilizing High-level Synthesis
Recent efforts for improving the performance of neural network (NN) accelerators that meet today’s application requirements have given rise to a new trend of logic-based NN inference relying on fixed function combinational logic. Mapping such large Boolean functions with many input variables and product terms to digital signal processors (DSPs) on Field-programmable gate arrays (FPGAs) needs a novel framework considering the structure and reconfigurability of DSP blocks during this process. The proposed methodology in this article maps the fixed function combinational logic blocks to a set of Boolean functions where Boolean operations corresponding to each function are mapped to DSP devices rather than look-up tables on the FPGAs to take advantage of the high performance, low latency, and parallelism of DSP blocks. This article also presents an innovative design and optimization methodology for compilation and mapping of NNs, utilizing fixed function combinational logic to DSPs on FPGAs employing high-level synthesis flow. Our experimental evaluations across several datasets and selected NNs demonstrate the comparable performance of our framework in terms of the inference latency and output accuracy compared to prior art FPGA-based NN accelerators employing DSPs.
- 14 October 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Applied Superconductivity
- Vol. 31  (9) , 1-11
A Logic Verification Framework for SFQ and AQFP Superconducting Circuits
Traditional logical equivalence checking (LEC) which plays a major role in the entire chip design process, faces challenges of meeting the requirements demanded by the many emerging technologies that are based on logic models different from the standard complementary metal oxide semiconductor (CMOS). In this paper, we propose a LEC framework to be employed in the verification process of superconducting electronics (SCE). Our LEC framework is compatible with the existing CMOS technologies and can also check features and capabilities that are unique to SCE. For instance, the performance of non-resistively biased Single-Flux Quantum (SFQ) circuits benefits from ultra-deep pipelining and verification of such circuits requires new models and algorithms. We, therefore, present the Multi-Cycle Input Dependency (MCID) circuit model which is a novel model representation of design to explicitly capture the dependency of primary outputs of the circuit on sequences of internal signals and inputs. Embedding the proposed circuit model and several structural checking modules, the process of verification can be independent of the underlying technology and signaling. We benchmark the proposed framework on post-synthesis SFQ and 4-phase adiabatic quantum flux parametron (AQFP) netlists. Results show a comparative verification time of SFQ circuit benchmark including 16-bit integer divider and ISCAS85 circuits with respect to the ABC tool for similar CMOS circuits.
- 14 October 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Applied Superconductivity
- Vol. 31  (9) , 1-11
A Logic Verification Framework for SFQ and AQFP Superconducting Circuits
Traditional logical equivalence checking (LEC) which plays a major role in the entire chip design process, faces challenges of meeting the requirements demanded by the many emerging technologies that are based on logic models different from the standard complementary metal oxide semiconductor (CMOS). In this paper, we propose a LEC framework to be employed in the verification process of superconducting electronics (SCE). Our LEC framework is compatible with the existing CMOS technologies and can also check features and capabilities that are unique to SCE. For instance, the performance of non-resistively biased Single-Flux Quantum (SFQ) circuits benefits from ultra-deep pipelining and verification of such circuits requires new models and algorithms. We, therefore, present the Multi-Cycle Input Dependency (MCID) circuit model which is a novel model representation of design to explicitly capture the dependency of primary outputs of the circuit on sequences of internal signals and inputs. Embedding the proposed circuit model and several structural checking modules, the process of verification can be independent of the underlying technology and signaling. We benchmark the proposed framework on post-synthesis SFQ and 4-phase adiabatic quantum flux parametron (AQFP) netlists. Results show a comparative verification time of SFQ circuit benchmark including 16-bit integer divider and ISCAS85 circuits with respect to the ABC tool for similar CMOS circuits.
- Open Access
NullaNet Tiny: Ultra-low-latency DNN Inference Through Fixed-function Combinational Logic
While there is a large body of research on efficient processing of deep neural networks (DNNs), ultra-low-latency realization of these models for applications with stringent, sub-microsecond latency requirements continues to be an unresolved, challenging problem. Field-programmable gate array (FPGA)-based DNN accelerators are gaining traction as a serious contender to replace graphics processing unit/central processing unit-based platforms considering their performance, flexibility, and energy efficiency. This paper presents NullaNet Tiny, an across-the-stack design and optimization framework for constructing resource and energy-efficient, ultra-low-latency FPGA-based neural network accelerators. The key idea is to replace expensive operations required to compute various filter/neuron functions in a DNN with Boolean logic expressions that are mapped to the native look-up tables (LUTs) of the FPGA device (examples of such operations are multiply-and-accumulate and batch normalization). At about the same level of classification accuracy, compared to Xilinx's LogicNets, our design achieves 2.36$\times$ lower latency and 24.42$\times$ lower LUT utilization.
Open AccessNullaNet Tiny: Ultra-low-latency DNN Inference Through Fixed-function Combinational Logic
While there is a large body of research on efficient processing of deep neural networks (DNNs), ultra-low-latency realization of these models for applications with stringent, sub-microsecond latency requirements continues to be an unresolved, challenging problem. Field-programmable gate array (FPGA)-based DNN accelerators are gaining traction as a serious contender to replace graphics processing unit/central processing unit-based platforms considering their performance, flexibility, and energy efficiency. This paper presents NullaNet Tiny, an across-the-stack design and optimization framework for constructing resource and energy-efficient, ultra-low-latency FPGA-based neural network accelerators. The key idea is to replace expensive operations required to compute various filter/neuron functions in a DNN with Boolean logic expressions that are mapped to the native look-up tables (LUTs) of the FPGA device (examples of such operations are multiply-and-accumulate and batch normalization). At about the same level of classification accuracy, compared to Xilinx's LogicNets, our design achieves 2.36$\times$ lower latency and 24.42$\times$ lower LUT utilization.