Abstract
This work presents a wired-logic (WL) and near-memory (NM) hybrid processor that enables general-purpose audio inference by embedding a compact self-supervised learning (SSL) pretrained encoder into an area-efficient WL core while retaining task-level adaptability through an NM-based classifier. To obtain a compact encoder suitable for WL implementation, we develop an NNN-SSL framework that trains a nonlinear neural network (NNN) using ternary weights, 80% pruning, and 14-bit activations within an augmentation-based SSL pipeline. The resulting 64-channel encoder provides unified feature extraction across speech, environmental sound, and musical-instrument domains. A global-pooling block and a frequency-serial fully connected (FC) architecture support variable-length audio inputs and reduce the FC area from 3.56mm2 to 0.76mm2. A 40 nm test chip achieves 48.1 TOPS/W for the convolution layer, while the full-network energy is estimated at 0.87 μJ/inference on GSCD-12. Furthermore, scaling the SSL pretraining dataset and the number of epochs enables accuracy improvements under a fixed-area constraint, showing that SSL can compensate for limited model capacity in WL-based designs. By combining SSL-driven general-purpose feature learning with WL-centric energy efficiency and minimal NM programmability, the proposed processor demonstrates a practical path toward compact, low-power, multi-domain audio AI for edge devices.
Funding Information
  • Japan Science and Technology Agency (JST)/Precursory Research for Embryonic Science and Technology (JPMJPR21B4)
  • JST Support for Pioneering Research Initiated by the Next Generation (JPMJSP2108)