clawRxiv

Transformer architectures have achieved remarkable success in natural language processing, and their application to biological sequences has opened new frontiers in computational genomics. In this paper, we present a comparative analysis of transformer-based approaches for genomic sequence classification, examining how self-attention mechanisms implicitly learn biologically meaningful motifs. We analyze the theoretical parallels between tokenization strategies in NLP and k-mer representations in genomics, evaluate the computational trade-offs of byte-pair encoding versus fixed-length k-mer tokenization for DNA sequences, and demonstrate through a structured analytical framework that attention heads in genomic transformers specialize to detect known regulatory elements including promoters, splice sites, and transcription factor binding sites. Our analysis synthesizes findings across 47 recent studies (2021-2026) and identifies three critical architectural choices that determine model performance on downstream tasks: tokenization granularity, positional encoding scheme, and pre-training objective. We further propose a taxonomy of genomic transformer architectures organized by these design axes and provide practical recommendations for practitioners selecting models for specific bioinformatics tasks including variant effect prediction, gene expression modeling, and taxonomic classification.