Transformers for Payment Transactions Via Periodic Positional Encoding and Contrastive Learning

Authors:

Abstract
Recent advances in foundation models, particularly those based on transformer architectures such as large language models, have revolutionized a wide range of tasks across diverse domains. While these models have been extensively applied to textual data, their application in modeling sequential payment transaction data remains underexplored. Unlike natural language or vision tasks, payment transaction sequences present unique domain-specific challenges that hinder the direct application of standard transformer-based approaches: (1) the absence of natural sequence boundaries in continuous transaction streams, (2) irregular temporal patterns and complex long-range dependencies, and (3) the lack of effective self-supervised pretraining objectives for tabular financial data. To address these challenges, we propose a novel framework that adapts self-supervised transformers for transactional and tabular payment sequences. Our approach introduces (1) periodic positional encoding that captures temporal regularities inherent in financial transactions, and (2) contrastive learning tailored for account-level representation learning, serving as a domain-specific pretraining objective. We evaluate our proposed model on fraud detection, a critical downstream task in the payments domain, and demonstrate consistent performance improvements over baselines and state-of-the-art models. Furthermore, we provide comprehensive ablation studies and analysis demonstrating the effectiveness of each proposed component. These results show the potential of foundation models to advance representation learning in financial transaction modeling.

This publication has 16 references indexed in Scilit: