Seyedarmin Azizi
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Seyedarmin Azizi has a total of 10 co-authors. Here are the top co-authors by number of shared publications:
- Massoud Pedram
University of Southern California, USA
6 shared publications - Erfan Baghaei Potraghloo
University of Southern California, USA
3 shared publications - Mahdi Nazemi
Department of Electrical & Computer Engineering University of Southern California Los Angeles CA USA
2 shared publications Massoud Pedram
2 shared publications- Mehdi Kamal
University of Southern California, USA
2 shared publications - Arash Fayyazi
University of Southern California Los Angeles California USA
1 shared publication
Latest Publications
- Open Access
- 3 November 2026
- Published by MDPI AG in Preprints.org
Reasoning for Generation, Generation for Reasoning: A Survey of Unified Multimodal Models
Unified multimodal models (UMMs) integrate visual understanding and image generation, creating the opportunity for each capability to strengthen the other. However, architectural unification does not guarantee such interaction: models frequently understand visual requirements that they fail to realize in generated images, and generate intermediate images that their reasoning does not use. This survey examines both directions of transfer and why each remains challenging. We review the architectural foundations and representative model families, and the tasks in which understanding and generation must interact. For understanding → generation, we organize methods into training-free approaches that improve inference with frozen components and training-based approaches that learn reasoning, alignment, verification, or refinement, and compare them by stage of intervention, required access, and computational cost. For generation → reasoning, we review visual-imagination methods that reason with generated images, latent visual tokens, geometric representations, drawing tools, or external world models. We further examine evaluation protocols that distinguish individual task performance, cross-modal consistency, and genuine synergy. Across both directions, a shared challenge emerges: a correct intermediate, whether a reasoning trace or an imagined image, is not necessarily used. We conclude with open problems in reasoning-to-generation alignment, verifiable imagination, reliable self-correction, and compute-matched evaluation.
- 3 November 2026
- Published by MDPI AG in Preprints.org
Open AccessReasoning for Generation, Generation for Reasoning: A Survey of Unified Multimodal Models
Unified multimodal models (UMMs) integrate visual understanding and image generation, creating the opportunity for each capability to strengthen the other. However, architectural unification does not guarantee such interaction: models frequently understand visual requirements that they fail to realize in generated images, and generate intermediate images that their reasoning does not use. This survey examines both directions of transfer and why each remains challenging. We review the architectural foundations and representative model families, and the tasks in which understanding and generation must interact. For understanding → generation, we organize methods into training-free approaches that improve inference with frozen components and training-based approaches that learn reasoning, alignment, verification, or refinement, and compare them by stage of intervention, required access, and computational cost. For generation → reasoning, we review visual-imagination methods that reason with generated images, latent visual tokens, geometric representations, drawing tools, or external world models. We further examine evaluation protocols that distinguish individual task performance, cross-modal consistency, and genuine synergy. Across both directions, a shared challenge emerges: a correct intermediate, whether a reasoning trace or an imagined image, is not necessarily used. We conclude with open problems in reasoning-to-generation alignment, verifiable imagination, reliable self-correction, and compute-matched evaluation.
- Open Access
- 15 October 2026
- Published by MDPI AG in Preprints.org
Looped Language Models: A Survey and Meta-Analysis of Recurrent-Depth Transformers
Looped language models apply one shared block of Transformer layers repeatedly to the same tokens. Their effective depth, the number of layer applications per token, can grow without adding parameters and can be chosen at inference time. Recurrent depth is thus a way to scale test-time compute in latent space instead of in generated tokens. The idea appears under many names, including looped Transformers, LoopLMs, recurrent-depth models, Universal Transformers, Mixture-of-Recursions, and recursive reasoning models such as HRM and TRM. In two years it has moved from synthetic tasks to open models such as Huginn and Ouro and to pretraining on up to 28 trillion tokens. According to press reports, it is also used in a frontier-lab model. Its literature has split across architecture, training, systems, theory and safety. We present the first survey dedicated to looped language models, built on a systematic corpus of 557 core papers screened from 7,323 candidates. We introduce a unified prelude–core–coda formalism that makes explicit how parameters, compute, memory and depth are counted. We organize the field from design, training and scaling to inference systems, theory, mechanisms, capabilities, other modalities and safety. A meta-analysis classifies 822 verified empirical claims by comparison regime. It shows that most apparent contradictions about whether “a recurrence is worth a layer” come from comparing parameter-matched with compute-matched results. The evidence fits one reading, in which a recurrence is worth less than a fresh block of parameters. Looping therefore helps when parameters are the limiting resource and hurts when it replaces them at fixed compute. We derive a research agenda from measured gaps, including the absence of work on robustness, alignment and privacy. We release the corpus, the extraction records with verbatim evidence for every number, and the claims ledger.
- 15 October 2026
- Published by MDPI AG in Preprints.org
Open AccessLooped Language Models: A Survey and Meta-Analysis of Recurrent-Depth Transformers
Looped language models apply one shared block of Transformer layers repeatedly to the same tokens. Their effective depth, the number of layer applications per token, can grow without adding parameters and can be chosen at inference time. Recurrent depth is thus a way to scale test-time compute in latent space instead of in generated tokens. The idea appears under many names, including looped Transformers, LoopLMs, recurrent-depth models, Universal Transformers, Mixture-of-Recursions, and recursive reasoning models such as HRM and TRM. In two years it has moved from synthetic tasks to open models such as Huginn and Ouro and to pretraining on up to 28 trillion tokens. According to press reports, it is also used in a frontier-lab model. Its literature has split across architecture, training, systems, theory and safety. We present the first survey dedicated to looped language models, built on a systematic corpus of 557 core papers screened from 7,323 candidates. We introduce a unified prelude–core–coda formalism that makes explicit how parameters, compute, memory and depth are counted. We organize the field from design, training and scaling to inference systems, theory, mechanisms, capabilities, other modalities and safety. A meta-analysis classifies 822 verified empirical claims by comparison regime. It shows that most apparent contradictions about whether “a recurrence is worth a layer” come from comparing parameter-matched with compute-matched results. The evidence fits one reading, in which a recurrence is worth less than a fresh block of parameters. Looping therefore helps when parameters are the limiting resource and hurts when it replaces them at fixed compute. We derive a research agenda from measured gaps, including the absence of work on robustness, alignment and privacy. We release the corpus, the extraction records with verbatim evidence for every number, and the claims ledger.
- Open Access
Activation Steering for Chain-of-Thought Compression
Large language models (LLMs) excel at complex reasoning when they include intermediate steps, known as "chains of thought" (CoTs). However, these rationales are often overly verbose, even for simple problems, leading to wasted context, increased latency, and higher energy consumption. We observe that verbose, English-heavy CoTs and concise, math-centric CoTs occupy distinct regions in the model's residual-stream activation space. By extracting and injecting a "steering vector" to transition between these modes, we can reliably shift generation toward more concise reasoning, effectively compressing CoTs without retraining. We formalize this approach as Activation-Steered Compression (ASC), an inference-time technique that shortens reasoning traces by directly modifying hidden representations. In addition, we provide a theoretical analysis of the impact of ASC on the output distribution, derived from a closed-form KL-divergence-bounded constraint to regulate steering strength. Using only 100 paired verbose and concise examples, ASC achieves up to 67.43% reduction in CoT length on MATH500 and GSM8K datasets, while maintaining accuracy across 7B, 8B, and 32B parameter models. As a training-free method, ASC introduces negligible runtime overhead and, on MATH500, delivers an average 2.73x speedup in end-to-end reasoning wall-clock time on an 8B model. This makes ASC a practical and efficient tool for streamlining the deployment of reasoning-capable LLMs in latency- or cost-sensitive settings. The code is available at: https://github.com/ArminAzizi98/ASC
Open AccessActivation Steering for Chain-of-Thought Compression
Large language models (LLMs) excel at complex reasoning when they include intermediate steps, known as "chains of thought" (CoTs). However, these rationales are often overly verbose, even for simple problems, leading to wasted context, increased latency, and higher energy consumption. We observe that verbose, English-heavy CoTs and concise, math-centric CoTs occupy distinct regions in the model's residual-stream activation space. By extracting and injecting a "steering vector" to transition between these modes, we can reliably shift generation toward more concise reasoning, effectively compressing CoTs without retraining. We formalize this approach as Activation-Steered Compression (ASC), an inference-time technique that shortens reasoning traces by directly modifying hidden representations. In addition, we provide a theoretical analysis of the impact of ASC on the output distribution, derived from a closed-form KL-divergence-bounded constraint to regulate steering strength. Using only 100 paired verbose and concise examples, ASC achieves up to 67.43% reduction in CoT length on MATH500 and GSM8K datasets, while maintaining accuracy across 7B, 8B, and 32B parameter models. As a training-free method, ASC introduces negligible runtime overhead and, on MATH500, delivers an average 2.73x speedup in end-to-end reasoning wall-clock time on an 8B model. This makes ASC a practical and efficient tool for streamlining the deployment of reasoning-capable LLMs in latency- or cost-sensitive settings. The code is available at: https://github.com/ArminAzizi98/ASC
- 12 May 2025
- Published by Springer Nature
- p. 55-66
Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy
- 12 May 2025
- Published by Springer Nature
- p. 55-66
Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy
- Open Access
- 5 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-6
PEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers
The deployment of Vision Transformers (ViTs) on hardware platforms, specially Field-Programmable Gate Arrays (FPGAs), presents many challenges, which are mainly due to the substantial computational and power requirements of their non-linear functions, notably layer normalization, softmax, and Gaussian Error Linear Unit (GELU). These critical functions pose significant obstacles to efficient hardware implementation due to their complex mathematical operations and the inherent resource count and architectural limitations of FPGAs. PEANO-ViT offers a novel approach to streamlining the implementation of the layer normalization layer by introducing a division-free technique that simultaneously approximates the division and square root function. Additionally, PEANO-ViT provides a multi-scale division strategy to eliminate division operations in the softmax layer, aided by a Padé-based approximation for the exponential function. Finally, PEANO-ViT introduces a piece-wise linear approximation for the GELU function, carefully designed to bypass the computationally intensive operations associated with GELU. In our comprehensive evaluations, PEANO-ViT exhibits minimal accuracy degradation ≤ 0.5% for DeiT-B) while significantly enhancing power efficiency, achieving improvements of 1.91X, 1.39X, and 8.01X for layer normalization, softmax, and GELU, respectively. This improvement is achieved through substantial reductions in DSP, LUT, and register counts for these non-linear operations. Consequently, PEANO-ViT enables efficient deployment of Vision Transformers on resource- and power-constrained FPGA platforms.
- 5 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-6
Open AccessPEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers
The deployment of Vision Transformers (ViTs) on hardware platforms, specially Field-Programmable Gate Arrays (FPGAs), presents many challenges, which are mainly due to the substantial computational and power requirements of their non-linear functions, notably layer normalization, softmax, and Gaussian Error Linear Unit (GELU). These critical functions pose significant obstacles to efficient hardware implementation due to their complex mathematical operations and the inherent resource count and architectural limitations of FPGAs. PEANO-ViT offers a novel approach to streamlining the implementation of the layer normalization layer by introducing a division-free technique that simultaneously approximates the division and square root function. Additionally, PEANO-ViT provides a multi-scale division strategy to eliminate division operations in the softmax layer, aided by a Padé-based approximation for the exponential function. Finally, PEANO-ViT introduces a piece-wise linear approximation for the GELU function, carefully designed to bypass the computationally intensive operations associated with GELU. In our comprehensive evaluations, PEANO-ViT exhibits minimal accuracy degradation ≤ 0.5% for DeiT-B) while significantly enhancing power efficiency, achieving improvements of 1.91X, 1.39X, and 8.01X for layer normalization, softmax, and GELU, respectively. This improvement is achieved through substantial reductions in DSP, LUT, and register counts for these non-linear operations. Consequently, PEANO-ViT enables efficient deployment of Vision Transformers on resource- and power-constrained FPGA platforms.
- 20 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Very Large Scale Integration (VLSI) Systems
- Vol. 32 (8) , 1414-1422
Low-Precision Mixed-Computation Models for Inference on Edge
This article presents a mixed-computation neural network processing approach for edge applications that incorporates low-precision (low-width) Posit and low-precision fixed point (FixP) number systems. This mixed-computation approach uses 4-bit Posit (Posit4), which has higher precision around 0, for representing weights with high sensitivity, while it uses 4-bit FixP (FixP4) for representing other weights. A heuristic for analyzing the importance and the quantization error of the weights is presented to assign the proper number system to different weights. In addition, a gradient approximation for Posit representation is introduced to improve the quality of weight updates in the backpropagation process. Due to the high energy consumption of the fully Posit-based computations, neural network operations are carried out in FixP or Posit/FixP. An efficient hardware implementation of an MAC operation with a first Posit operand and FixP for a second operand and accumulator is presented. The efficacy of the proposed low-precision mixed-computation approach is extensively assessed on vision and language models. The results show that on average, the accuracy of the mixed-computation is about 1.5% higher than that of FixP with a cost of 0.19% energy overhead.
- 20 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Very Large Scale Integration (VLSI) Systems
- Vol. 32 (8) , 1414-1422
Low-Precision Mixed-Computation Models for Inference on Edge
This article presents a mixed-computation neural network processing approach for edge applications that incorporates low-precision (low-width) Posit and low-precision fixed point (FixP) number systems. This mixed-computation approach uses 4-bit Posit (Posit4), which has higher precision around 0, for representing weights with high sensitivity, while it uses 4-bit FixP (FixP4) for representing other weights. A heuristic for analyzing the importance and the quantization error of the weights is presented to assign the proper number system to different weights. In addition, a gradient approximation for Posit representation is introduced to improve the quality of weight updates in the backpropagation process. Due to the high energy consumption of the fully Posit-based computations, neural network operations are carried out in FixP or Posit/FixP. An efficient hardware implementation of an MAC operation with a first Posit operand and FixP for a second operand and accumulator is presented. The efficacy of the proposed low-precision mixed-computation approach is extensively assessed on vision and language models. The results show that on average, the accuracy of the mixed-computation is about 1.5% higher than that of FixP with a cost of 0.19% energy overhead.
- Open Access
Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy
As Vision Transformers (ViTs) increasingly set new benchmarks in computer vision, their practical deployment on inference engines is often hindered by their significant memory bandwidth and (on-chip) memory footprint requirements. This paper addresses this memory limitation by introducing an activation-aware model compression methodology that uses selective low-rank weight tensor approximations of different layers to reduce the parameter count of ViTs. The key idea is to decompose the weight tensors into a sum of two parameter-efficient tensors while minimizing the error between the product of the input activations with the original weight tensor and the product of the input activations with the approximate tensor sum. This approximation is further refined by adopting an efficient layer-wise error compensation technique that uses the gradient of the layer's output loss. The combination of these techniques achieves excellent results while it avoids being trapped in a shallow local minimum early in the optimization process and strikes a good balance between the model compression and output accuracy. Notably, the presented method significantly reduces the parameter count of DeiT-B by 60% with less than 1% accuracy drop on the ImageNet dataset, overcoming the usual accuracy degradation seen in low-rank approximations. In addition to this, the presented compression technique can compress large DeiT/ViT models to have about the same model size as smaller DeiT/ViT variants while yielding up to 1.8% accuracy gain. These results highlight the efficacy of our approach, presenting a viable solution for embedding ViTs in memory-constrained environments without compromising their performance.
Open AccessMemory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy
As Vision Transformers (ViTs) increasingly set new benchmarks in computer vision, their practical deployment on inference engines is often hindered by their significant memory bandwidth and (on-chip) memory footprint requirements. This paper addresses this memory limitation by introducing an activation-aware model compression methodology that uses selective low-rank weight tensor approximations of different layers to reduce the parameter count of ViTs. The key idea is to decompose the weight tensors into a sum of two parameter-efficient tensors while minimizing the error between the product of the input activations with the original weight tensor and the product of the input activations with the approximate tensor sum. This approximation is further refined by adopting an efficient layer-wise error compensation technique that uses the gradient of the layer's output loss. The combination of these techniques achieves excellent results while it avoids being trapped in a shallow local minimum early in the optimization process and strikes a good balance between the model compression and output accuracy. Notably, the presented method significantly reduces the parameter count of DeiT-B by 60% with less than 1% accuracy drop on the ImageNet dataset, overcoming the usual accuracy degradation seen in low-rank approximations. In addition to this, the presented compression technique can compress large DeiT/ViT models to have about the same model size as smaller DeiT/ViT variants while yielding up to 1.8% accuracy gain. These results highlight the efficacy of our approach, presenting a viable solution for embedding ViTs in memory-constrained environments without compromising their performance.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 9635-9646
LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation
Seyedarmin Azizi, Souvik Kundu, Massoud Pedram. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 9635-9646
Open AccessLaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation
Seyedarmin Azizi, Souvik Kundu, Massoud Pedram. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.