Sequence length scaling in vision transformers for scientific images on frontier
- 19 November 2025
- journal article
- Published by SAGE Publications in The International Journal of High Performance Computing Applications
- Vol. 40 (3) , 273-290
Abstract
Vision Transformers (ViTs) are pivotal for foundational models in scientific imagery, including Earth science applications, due to their capability to process large sequence lengths. While transformers for text have inspired scaling sequence lengths in ViTs, adapting these for ViTs introduces unique challenges. We develop distributed sequence parallelism for ViTs, enabling them to handle up to 1M tokens. Our approach, leveraging DeepSpeed-Ulysses and Long-Sequence-Segmentation with model sharding, is the first to apply sequence parallelism in ViT training, achieving a 94% batch scaling efficiency on 2,048 AMD-MI250X GPUs. Evaluating sequence parallelism in ViTs, particularly in models up to 10B parameters, highlighted substantial bottlenecks. We countered these with hybrid sequence, pipeline, and flash attention strategies, to scale beyond single GPU memory limits. Our method significantly enhances climate modeling accuracy by 20% in temperature predictions, marking the first training of a vision transformer model to convergence with a sequence length of 188K tokens, using full self-attention.Keywords
All Related Versions
Funding Information
- UT-Battelle (DE-AC05-00OR22725)
This publication has 19 references indexed in Scilit:
- One for All: Toward Unified Foundation Models for Earth Vision2024
- Comparative Study of Large Language Model Architectures on Frontier2024
- FuXi: a cascade machine learning forecasting system for 15-day global weather forecastnpj Climate and Atmospheric Science, 2023
- A high-resolution large-scale dataset of pathological and normal white blood cellsScientific Data, 2023
- Accurate medium-range global weather forecasting with 3D neural networksNature, 2023
- A Large-scale Synthetic Pathological Dataset for Deep Learning-enabled Segmentation of Breast CancerScientific Data, 2023
- Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning2022
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- PAIP 2019: Liver cancer segmentation challengeMedical Image Analysis, 2021
- ImageNet: A large-scale hierarchical image database2009