Abstract
This paper introduces a novel approach to sequential user modeling by developing distilled sequence model architectures for specific user clusters. While generic, interest-based models have become a cornerstone of personalized recommendation systems, they often fail to provide high-quality recommendations for “power users” with highly specific interests, as their recommendations can be diluted by broader user behavior. To address this, we present a framework for creating specialized transformer-based sequence models tailored to individual domains, such as horror movies, music, and sports. Our method, built upon the principles of the Pinner-Former architecture, leverages domain-focused training datasets to generate distilled user embeddings. This approach offers several advantages, including enabling highly focused retrieval and requiring less computational resources, which results in a smaller, more efficient model that's easier to deploy in a production environment. However, this method also presents challenges, such as the difficulty of acquiring high-quality domain-specific data and a lack of scalability for a large number of verticals, as a separate model is needed for each domain. We demonstrate a 19.9% improvement in Recall@10 with 70-75% computational efficiency improvements, showing that this specialized approach significantly improves personalization for power users while enabling more efficient deployment.