Free Energy Mixer
Preprint
Abstract
Standard attention stores keys/values losslessly but reads them via a per-head convex average, blocking channel-wise selection. We propose the Free Energy Mixer (FEM): a free-energy (log-sum-exp) read that applies a value-driven, per-channel log-linear tilt to a fast prior (e.g., from queries/keys in standard attention) over indices. Unlike methods that attempt to improve and enrich the $(q,k)$ scoring distribution, FEM treats it as a prior and yields a value-aware posterior read at unchanged complexity, smoothly moving from averaging to per-channel selection as the learnable inverse temperature increases, while still preserving parallelism and the original asymptotic complexity ($O(T^2)$ for softmax; $O(T)$ for linearizable variants). We instantiate a two-level gated FEM that is plug-and-play with standard and linear attention, linear RNNs and SSMs. It consistently outperforms strong baselines on NLP, vision, and time-series at matched parameter budgets.
Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- SCFormer: Structured Channel-Wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting2026
- ACCESS: Advancing Innovation2023
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them2023
- Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers2023
- Memory-efficient Transformers via Top-k Attention2021
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleProceedings of the AAAI Conference on Artificial Intelligence, 2020
- HellaSwag: Can a Machine Really Finish Your Sentence?2019
- Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together2019
- Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks2018
- Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent NetworksNeural Computation, 1992