Enterprise AI Analysis

Safety Alignment in LLMs

The foundational paradigm for LLM safety emerged from Reinforcement Learning from Human Feedback (RLHF), as established by InstructGPT. Subsequent works like Llama 2 refined this with dual reward models and extensive red-teaming. Constitutional AI (CAI) introduced Reinforcement Learning from AI Feedback (RLAIF) where models critique their own outputs. More recently, Direct Preference Optimization (DPO) simplified this pipeline by optimizing the policy directly from preference data, eliminating the need for explicit reward modeling.

A robust ecosystem of safety benchmarks has emerged alongside these methods, evaluating models across universal dimensions like toxicity, fairness, and privacy. However, a critical limitation persists: safety is defined at the population level. Existing alignment techniques and evaluation frameworks treat safety as a universal constraint, blocking objectively harmful content for all users identically. They lack the granularity to address personalized safety sensitivities, which vary across users and contexts.

LLM-based Conversational Recommender Systems (CRS)

Early CRS relied heavily on structured knowledge, with works like ReDial introducing human-annotated datasets and approaches like KBRD, KGSF, and UniCRS utilizing Knowledge Graphs (KGs) and Graph Neural Networks (GNNs). The advent of LLMs transformed this landscape by enabling more flexible, training-free interaction, leveraging generative and reasoning capabilities.

Methods like Chat-Rec leverage LLMs as interactive agents, converting user profiles into prompts. InstructRec and TALLRec treat recommendation as an instruction-following task, demonstrating the effectiveness of fine-tuning smaller LLMs. Despite these advances in accuracy, recommendation safety remains a critical blind spot, with existing methods not addressing content safety at the individual level.

Introducing SAFEREC: A User-Centric Safety Benchmark

To bridge the gap in personalized safety, we introduce SAFEREC, the first user-centric safety analysis benchmark for CRS. SAFEREC augments the Reddit-V2 conversational movie recommendation dataset with explicit safety annotations, establishing the movie (SAFEMOVIE) and game (SAFEGAME) domains as generalizable case studies.

SAFEREC operationalizes personalized safety through a fine-grained representation of user sensitivities, introducing the notion of Latent Traits. These user-specific sensitivity profiles (e.g., history of self-harm, strict aversion to sexual violence, or phobia of needles) directly map to structured content metadata.

SAFEREC Safety Knowledge Base Details

SAFEMOVIE Oracle: Fuses DoesTheDogDie (DDD) and IMDb Parent Guide (IPG) data. IPG provides severity annotations for five coarse dimensions: {Sex/Nudity, Violence/Gore, Profanity, Alcohol/Drugs/Smoking, Frightening/Intense Scenes}. DDD offers 137 fine-grained warning tags. These are grouped via LLM-guided clustering into 20 explicit user sensitivity traits (e.g., Anti-gore, Kid-safety).

A continuous parental-guidance risk is computed as a normalized weighted sum: pg_risk(m, t) = w(t)s(m) / (3 Σi Wi(t)) ∈ [0, 1]. Hard triggers from DDD are incorporated for specific concerns, leading to a final trait-conditioned risk: final_risk(m, t) = max(pg_risk(m, t), trigger(m, t)).

SAFEGAME Oracle: Uses Entertainment Software Rating Board (ESRB) data with categorical age ratings and content descriptors. 10 game-domain sensitivity traits (e.g., Anti-gore, Extreme Violence) are defined. Final risk score: final_risk(g, t) = trigger(g, t) · α(ρ(g)).

SafeCRS: A Safety-Aware Training Framework

Building on the personalized safety challenges, we propose SAFECRS, a two-stage pipeline designed to jointly optimize recommendation quality and personalized safety alignment. It integrates Safe Supervised Fine-Tuning (Safe-SFT) with Safe Group reward-Decoupled Normalization Policy Optimization (Safe-GDPO).

Stage 1: Safe-SFT (Supervised Fine-Tuning)

Safe-SFT explicitly teaches the model to perform a safety analysis of candidate items and produce a final recommendation list that excludes unsafe items. It starts with base recommendations (e.g., from GPT-40) and uses an external safety database to score items under fixed latent traits.

Training targets are structured into two parts: a safe reasoning block, which documents detected preferences, filtered items with rationales/risk scores, and safe item counts; and a solution block, containing only the final safe recommendations. This ensures the model learns to justify removals without hallucinating additional harms.

Stage 2: Safe-GDPO (Group reward-Decoupled Normalization Policy Optimization)

Safe-GDPO refines rank-wise recommendations by further updating the policy, improving recommendation quality while preserving user-aware safety. It addresses sparsity levels of different reward signals using per-reward normalization before aggregation.

Three independent reward functions are optimized: relevance (binary hits for ground-truth matches), safety (rank-discounted penalties for unsafe items), and output-length compliance (scalar count reward). By decoupling and normalizing each reward dimension independently, GDPO preserves informative gradient signals from sparse rewards, preventing them from being overwhelmed by denser safety and format rewards.

Experimental Results: Safety and Relevance Trade-off

Experiments on the SAFEREC benchmark (SAFEMOVIE and SAFEGAME) demonstrate that existing CRS methods, regardless of model scale or architecture, do not adequately respect user-specific safety constraints, resulting in universally high violation rates.

SafeCRS vs. State-of-the-Art Baselines

SafeCRS variants drastically reduce safety violations while maintaining competitive recommendation quality. For instance, on SAFEMOVIE, Llama-3.1-8B with SafeCRS achieves Recall@10 = 0.1111 and NDCG@10 = 0.0737 (comparable to GPT-5.2), while reducing SVR@5 from 0.3508 to 0.0122 (a 96.5% relative reduction). Even the smallest backbone (Qwen2.5-0.5B) achieves near-zero violation rates.

The results consistently show that SafeCRS models shift towards the Pareto frontier, optimizing both safety and relevance simultaneously, demonstrating strong cross-domain generalizability on both movie and game domains.

Impact of Safe-SFT and Safe-GDPO (Ablation Study)

Effect of Safe-SFT: Safe-SFT provides the foundational improvement over zero-shot baselines. On SAFEMOVIE, Llama-3.1-8B improves Recall@5 by +47.3% while reducing SVR@5 by -89.7%. This stage trains the model to follow catalog constraints and produce safety-aware reasoning.

Effect of Safe-GDPO: Building on Safe-SFT, Safe-GDPO further tightens the safety-relevance Pareto frontier through per-reward normalization. On SAFEMOVIE, Qwen2.5-0.5B sees SVR@5 drop by -93.8% while Recall@5 simultaneously improves by +56.7%. This demonstrates that safety and relevance are not inherently at odds when rewards are properly decoupled.

Conclusion & Future Outlook

This work identified personalized safety alignment as a critical yet underexplored challenge in LLM-based Conversational Recommender Systems (CRS). We addressed this by introducing SAFEREC, the first user-centric safety benchmark dataset, designed to systematically evaluate safety failures under user-specific constraints.

Our proposed framework, SAFECRS, effectively integrates Safe-SFT with Safe-GDPO to jointly prioritize recommendation relevance and individual safety sensitivities. Extensive experiments across movie and game domains demonstrate that SAFECRS drastically reduces safety violations by up to 96.5% while maintaining or exceeding the recommendation quality of state-of-the-art baselines.

By establishing a domain-agnostic approach to safety reasoning and reward decoupling, our work provides a robust foundation for building trustworthy conversational agents that respect user-specific content suitability constraints. This represents a significant step towards more responsible and user-centric AI in recommendation.

Code, benchmark, and trained checkpoints are available at https://github.com/xxyyffyeah/SafeCRS to support reproducibility.