Soroush Abbasi Koohpayegani
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Soroush Abbasi Koohpayegani has a total of 6 co-authors. Here are the top co-authors by number of shared publications:
- Hamed Pirsiavash
University of California Davis
3 shared publications - Ali Abbasi
Department of Computer Science, Vanderbilt University, Nashville, TN, USA
1 shared publication - K. L. Navaneet
University of California
1 shared publication - Kossar Pourahmadi Meibodi
University of California
1 shared publication - Parsa Nooralinejad
CS Department, University of California, Davis, CA, USA
1 shared publication - Soheil Kolouri
Department of Computer Science Vanderbilt University
1 shared publication
Latest Publications
- 23 November 2024
- Published by Springer Nature
- p. 330-349
CompGS: Smaller and Faster Gaussian Splatting with Vector Quantization
- 23 November 2024
- Published by Springer Nature
- p. 330-349
CompGS: Smaller and Faster Gaussian Splatting with Vector Quantization
- Open Access
SimA: Simple Softmax-free Attention for Vision Transformers
Recently, vision transformers have become very popular. However, deploying them in many applications is computationally expensive partly due to the Softmax layer in the attention block. We introduce a simple but effective, Softmax-free attention block, SimA, which normalizes query and key matrices with simple $\ell_1$-norm instead of using Softmax layer. Then, the attention block in SimA is a simple multiplication of three matrices, so SimA can dynamically change the ordering of the computation at the test time to achieve linear computation on the number of tokens or the number of channels. We empirically show that SimA applied to three SOTA variations of transformers, DeiT, XCiT, and CvT, results in on-par accuracy compared to the SOTA models, without any need for Softmax layer. Interestingly, changing SimA from multi-head to single-head has only a small effect on the accuracy, which simplifies the attention block further. The code is available here: https://github.com/UCDvision/sima
Open AccessSimA: Simple Softmax-free Attention for Vision Transformers
Recently, vision transformers have become very popular. However, deploying them in many applications is computationally expensive partly due to the Softmax layer in the attention block. We introduce a simple but effective, Softmax-free attention block, SimA, which normalizes query and key matrices with simple $\ell_1$-norm instead of using Softmax layer. Then, the attention block in SimA is a simple multiplication of three matrices, so SimA can dynamically change the ordering of the computation at the test time to achieve linear computation on the number of tokens or the number of channels. We empirically show that SimA applied to three SOTA variations of transformers, DeiT, XCiT, and CvT, results in on-par accuracy compared to the SOTA models, without any need for Softmax layer. Interestingly, changing SimA from multi-head to single-head has only a small effect on the accuracy, which simplifies the attention block further. The code is available here: https://github.com/UCDvision/sima
- 30 November 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 30 (10709908) , 1812-1816
Multi-Agent Lifelong Implicit Neural Learning
Implicit neural representations (INRs) have emerged as powerful tools for the continuous representation of signals, finding applications in imaging, computer graphics, and signal compression. Additionally, decentralized multi-agent systems are crucial in various applications, frequently leading to enhanced reliability and efficiencies in computation and communication. In this paper, we explore using multi-agent Lifelong Learning (LL) systems for learning INRs. We propose a rigorous problem setup and evaluation plan to investigate the efficacy of such systems compared to single-agent and multi-task learning baselines. Our research, conducted across varied dimensions, demonstrates promising results, thereby contributing a novel perspective to the realm of continual learning.
- 30 November 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 30 (10709908) , 1812-1816
Multi-Agent Lifelong Implicit Neural Learning
Implicit neural representations (INRs) have emerged as powerful tools for the continuous representation of signals, finding applications in imaging, computer graphics, and signal compression. Additionally, decentralized multi-agent systems are crucial in various applications, frequently leading to enhanced reliability and efficiencies in computation and communication. In this paper, we explore using multi-agent Lifelong Learning (LL) systems for learning INRs. We propose a rigorous problem setup and evaluation plan to investigate the efficacy of such systems compared to single-agent and multi-task learning baselines. Our research, conducted across varied dimensions, demonstrates promising results, thereby contributing a novel perspective to the realm of continual learning.