Research & Analysis

Related Work

This section introduces the related work from three aspects: RLHF, which serves as the basis of the single-dimensional model, decoding-time alignment, and Multi-dimensional Personalization.

Reinforcement Learning from Human Feedback.

RLHF [1, 2, 9, 23, 37] serves as the dominant paradigm for aligning LLMs with human preferences. It traditionally employs a three-stage pipeline: Supervised Fine-Tuning (SFT), training a reward model from human feedback, and then enhancing the policy through a reinforcement learning algorithm, most notably PPO [27]. More recent approaches, such as DPO [25], offer an alternative to this final stage by directly optimizing the policy on preference data, bypassing the need for an explicit reward model and RL optimization.

Decoding-time Alignment.

Decoding-time alignment methods emerged as an effective approach to satisfy diverse user requirements, which can adapt to different objectives without retraining. PAD [5] introduces token-level personalized rewards by decoupling text generation dynamics from user preferences, enabling dynamic adjustment of base model predictions during inference. It leverages a single policy and reward model to generalize to unseen preferences. MOD [28] employs a closed-form solution derived via Legendre transformation to combine predictions from multiple base models-each aligned to distinct objectives-during decoding. By weighting model outputs algebraically, MOD achieves precise control over objective weights. Drift [18] extends this direction by inferring user preferences through attribute-level decomposition and combining attribute signals directly at decoding time, enabling effective few-shot personalization without explicit reward models.

Multi-dimensional Personalization.

For the more complex multi-dimensional personalization alignment challenge of having models satisfy multiple (or even conflicting) objectives simultaneously (e.g., balancing usefulness and harmlessness, or simulating a specific MBTI/Role-Playing persona), the research community has explored different strategies. The main approaches include training a single model to optimize a weighted multi-objective function (e.g., MORLHF, MODPO [36]), combining the parameters of independently-trained single-preference models (e.g., DPO Soups [15]) during decoding, and bootstrapping by combining reward signals or model predictions at decoding (e.g., PAD [5], MOD [28]). Drift [18] decomposes user preferences into interpretable attributes and composes them at decoding time, providing a new perspective on multi-dimensional personalization.

Methodology

This section details the MPG framework by first formally defining the multi-personality generation problem, then establishing the theoretical underpinnings of our method based on the density ratio principle, and finally introducing our efficient implementation, Speculative Chunk-level based Rejection sampling (SCR).

Problem Definition and Formulation

Our paper addresses the problem of generating text that exhibits multiple personality attributes at decode time for LLMs. Specifically, given an input x and preference weight vector a = [α₁,..., αν] (with constraints a₁ ≥ 0 and ∑i=1 α₁ = 1), generate a text sequence y that simultaneously embodies these personality features under a set of N desired personality attributes {d₁,..., dn}.

MPG Based on Density Ratio

The core principle guiding our approach stems from a key feature observed in LLM preference alignment training (e.g., DPO [25]): the probability density ratio of an optimized policy model πο (y|x) relative to the reference model ref(yx) used in its training, r(y|x) = πρ(yx) (yx), implicitly encodes the preference information learned by the model. This ratio quantifies the model's preference for a particular output sequence y compared to its likelihood under ref(y|x). We take advantage of this principle to sample from our target multi-personality distribution πMPG(y|x).

SCR Algorithm

We implement MPG in decoding-time algorithm SCR (Speculative Chunk-level Rejection Sampling).

Unlike sequence-level rejection sampling, SCR proposes and validates k-token chunks at once, integrating speculative decoding [20]. By validating multi-token speculative proposals against the aggregated density ratio, SCR preserves the correctness of rejection sampling while amortizing expensive target-policy validation over longer spans, reducing the frequency of preference model evaluations and enabling efficient parallelization.