Deep Hypercomplex Networks for Spatiotemporal Data Processing: Parameter efficiency and superior performance
- 20 August 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Magazine
- Vol. 41 (3) , 101-112
Abstract
Hypercomplex numbers, such as quaternions and octonions, have recently gained attention because of their advantageous properties over real numbers, e.g., in the development of parameter-efficient neural networks. For instance, the 16-component sedenion has the capacity to reduce the number of network parameters by a factor of 16. Moreover, hypercomplex neural networks offer advantages in the processing of spatiotemporal data as they are able to represent variable temporal data divisions through the hypercomplex components. Similarly, they support multimodal learning, with each component representing an individual modality. In this article, the key components of deep learning in the hypercomplex domain are introduced, encompassing concatenation, activation functions, convolution, and batch normalization. The use of the backpropagation algorithm for training hypercomplex networks is discussed in the context of hypercomplex algebra. These concepts are brought together in the design of a ResNet backbone using hypercomplex convolution, which is integrated within a U-Net configuration and applied in weather and traffic forecasting problems. The results demonstrate the superior performance of hypercomplex networks compared to their real-valued counterparts, given a fixed parameter budget, highlighting their potential in spatiotemporal data processing.Keywords
This publication has 19 references indexed in Scilit:
- Multimodal learning with graphsNature Machine Intelligence, 2023
- Wind Speed Forecasting Using the Stationary Wavelet Transform and Quaternion Adaptive-Gradient MethodsIEEE Access, 2021
- Metacognitive Sedenion-Valued Neural Network and its Learning AlgorithmIEEE Access, 2020
- Deep octonion networksNeurocomputing, 2020
- Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text2020
- Quaternion Convolutional Neural Networks for Heterogeneous Image Processing2019
- CondenseNet: An Efficient DenseNet Using Learned Group Convolutions2018
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- Cognitive Quaternion Valued Neural Network and some applicationsNeurocomputing, 2017
- Complex domain backpropagationIEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing, 1992