Compressing Large Language Models with PCA Without Performance Loss
Preprint
Abstract
We demonstrate that Principal Component Analysis (PCA), when applied in a structured manner, either to polar-transformed images or segment-wise to token sequences, enables extreme compression of neural models without sacrificing performance. Across three case studies, we show that a one-layer classifier trained on PCA-compressed polar MNIST achieves over 98 percent accuracy using only 840 parameters. A two-layer transformer trained on 70-dimensional PCA-reduced MiniLM embeddings reaches 76.62 percent accuracy on the 20 Newsgroups dataset with just 81000 parameters. A decoder-only transformer generates coherent token sequences from 70-dimensional PCA embeddings while preserving over 97 percent cosine similarity with full MiniLM representations, using less than 17 percent of the parameter count of GPT-2. These results highlight PCA-based input compression as a general and effective strategy for aligning model capacity with information content, enabling lightweight architectures across multiple modalities.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Exploring Dimensionality Reduction Techniques in Multilingual TransformersCognitive Computation, 2022
- Knowledge Base Index Compression via Dimensionality and Precision Reduction2022
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Single Training Dimension Selection for Word Embedding with PCA2019
- Principal component analysis: a review and recent developmentsPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2016
- Eigenfaces for RecognitionJournal of Cognitive Neuroscience, 1991
- Learning representations by back-propagating errorsNature, 1986
- Analysis of a complex of statistical variables into principal components.Journal of Educational Psychology, 1933
- LIII. On lines and planes of closest fit to systems of points in spaceJournal of Computers in Education, 1901