RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling

Ziyuan Wang; Bohao Tang; Fei Zhang; Shuo Han; Pengfei Liu

title

authors

Ziyuan Wang, Bohao Tang, Fei Zhang, Shuo Han, Pengfei Liu

date

2026-08-24

venue

arXiv (preprint, not peer reviewed)

arXiv:2608.22849

affiliation

SII-GAIR

upvotes

3 (Hugging Face Daily Papers, 2026-11-01)

links

abs pdf hf code

abstract

Full-length RNAs, particularly messenger RNAs, often exceed the context lengths used to pretrain existing RNA foundation models, limiting complete-transcript modeling at single-nucleotide resolution. We present RIBOSPAN, a 1.61-billion-parameter bidirectional RNA foundation model natively pretrained with context lengths up to 10,240 nt. RIBOSPAN combines dense bidirectional self-attention, single-nucleotide tokenization, and attention-isolated sequence packing to enable high-resolution modeling of complete long RNAs. We evaluate the model through nucleotide reconstruction, a controlled long-context representation benchmark, and frozen RNA-type representation analysis. Native 10K pretraining preserves strong reconstruction at 10,240 tokens, while continued pretraining with 40% masking improves recovery under heavy corruption while preserving representation quality. The long-context benchmark further shows that native 10K models maintain strong contextual responsiveness and context-specific representation separation while keeping perturbation-induced representation changes highly localized. Inference-time YaRN scaling recovers much of the contextual organization lost by direct extrapolation of short-context models, but induces substantially greater distal representation diffusion. Frozen-representation evaluations further demonstrate state-of-the-art RNA representation quality, with RIBOSPAN achieving the strongest overall performance across diverse RNA types and retaining a clear advantage on long RNAs. Building on the same backbone, we develop a multidimensionally conditioned discrete-diffusion framework for full-length mRNA generation and redesign, including synonymous-codon diffusion for protein-preserving CDS optimization. Together, RIBOSPAN establishes a powerful long-context foundation for transferable RNA representation learning and full-transcript mRNA design.

@misc{wang2026ribospan,
  title={RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling},
  author={Ziyuan Wang and Bohao Tang and Fei Zhang and Shuo Han and Pengfei Liu},
  year={2026},
  eprint={2608.22849},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2608.22849}
}