MixFormer: Co-Scaling Up Dense and Sequence in Industrial Recommenders
Open Access
- 8 August 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 7422-7431
Abstract
As industrial recommender systems enter a scaling-driven regime, Transformer architectures have become increasingly attractive for scaling models towards larger capacity and longer sequence. However, existing Transformer-based recommendation models remain structurally fragmented, where sequence modeling and feature interaction are implemented as separate modules with independent parameterization. Such designs introduce a fundamental co-scaling challenge, as model capacity must be suboptimally allocated between dense feature interaction and sequence modeling under a limited computational budget. In this work, we propose MixFormer, a unified Transformer-style architecture tailored for recommender systems, which jointly models sequential behaviors and feature interactions within a single backbone. Through a unified parameterization, MixFormer enables effective co-scaling across both dense capacity and sequence length, mitigating the trade-off observed in decoupled designs. Moreover, the integrated architecture facilitates deep interaction between sequential and non-sequential representations, allowing high-order feature semantics to directly inform sequence aggregation and enhancing overall expressiveness. To ensure industrial practicality, we further introduce a user-item decoupling strategy for efficiency optimizations that significantly reduce redundant computation and inference latency. Extensive experiments on large-scale industrial datasets demonstrate that MixFormer consistently exhibits superior accuracy and efficiency. Furthermore, large-scale online A/B tests on two production recommender systems, Douyin and Douyin Lite, show consistent improvements in user engagement metrics, including active days and in-app usage duration.Keywords
This publication has 20 references indexed in Scilit:
- TWIN: TWo-stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou2023
- Learning to Retrieve User Behaviors for Click-through Rate EstimationACM Transactions on Information Systems, 2023
- Sampling Is All You Need on Modeling Long-Term User Behaviors for CTR Prediction2022
- DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems2021
- Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction2020
- Behavior sequence transformer for e-commerce recommendation in Alibaba2019
- Deep Interest Network for Click-Through Rate Prediction2018
- xDeepFM2018
- Deep & Cross Network for Ad Click Predictions2017
- Factorization Machines2010