Protein Autoregressive Modeling via Multiscale Structure Generation

PAR Framework: Multi-scale Structure Generation

Building on this multi-scale framework, PAR includes three key components (Fig. 1). The multi-scale downsampling creates coarse-to-fine structural representations to serve as structural context and targets during training. AR transformer, a stack of non-equivariant attention layers [42], encodes all preceding scales to produce a scale-wise conditional embedding following Li et al. [30].

The flow-based backbone decoder is conditioned on this embedding to model Ca backbone atoms directly. As a result, PAR avoids both discretization of protein structures and residue-wise unidirectional autoregressive ordering, thereby overcoming the two aforementioned limitations that compromise structural fidelity and generative quality.

Moreover, training on ground-truth structural context, AR models suffer from exposure bias [3], which is a key challenge substantially reducing structure generation quality in our preliminary study. We effectively mitigate such issue via noisy context learning and scheduled sampling, allowing the model to learn from corrupted context.

Performance and Zero-Shot Generalization

We begin by evaluating PAR on unconditional backbone generation and compare it with existing structure generative methods in §4.1. Next, we examine its zero-shot generalization ability in §4.2. We then study the scaling behavior, efficient sampling, and propose strategies to mitigate exposure bias, along with additional ablations in §4.3.

For unconditional generation, PAR exhibits favorable scaling behavior, yielding competitive results on distributional metrics like Fréchet Protein Structure Distance (FPSD). Unlike diffusion models, which operate at a single scale, PAR flexibly handles inputs at various granularities, and hence shows zero-shot generalization in tasks like prompt-based generation and motif scaffolding.

The multi-scale formulation enables PAR to orchestrate sampling strategies, achieving a 2.5x sampling speedup compared to single-scale baselines. Finally, PAR provides a more general framework, incorporating flow-based models as a special case when restricted to a single scale, and thus remains compatible with techniques from flow-based models like self-conditioning [9].

Conclusion and Future Directions

PAR is the first multi-scale autoregressive model for protein backbone generation, offering a general framework that includes flow-based methods as a special case. PAR addressed limitations of standard autoregressive models, such as unidirectional dependency, discretization, and exposure bias. Our method robustly models structures over multiple granularities and in turn enables strong zero-shot generalization.

We hope that PAR unlocks the potential of autoregressive modeling for protein design. Some promising open directions include: (1) Conformational dynamics modeling. PAR can, in principle, perform zero-shot modeling of conformational distributions: we downsample a structure and upsample it with PAR to mimic local molecular dynamics. (2) All-atom modeling. This work focuses on backbone Ca atoms to prioritize autoregressive design, but it's natural to extend to full-atom representations [37].