Personalized Federated Learning
Co-LORA: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients
Authors: Minhyuk Seo, Taeheon Kim, Hankook Lee, Jonghyun Choi, Tinne Tuytelaars
As AI becomes more personal, e.g., Agentic AI, there is an increasing need for personalizing models for various use cases. Personalized federated learning (PFL) enables each client to collaboratively leverage other clients' knowledge for better adaptation to the task of interest, without privacy risks. Despite its potential, existing PFL methods remain confined to rather simplified scenarios where data and models are the same across clients. To move towards realistic scenarios, we move beyond these restrictive assumptions by addressing both data and model heterogeneity. We propose a task-relevance-aware model aggregation strategy to reduce parameter interference under heterogeneous data. Moreover, we introduce Co-LoRA, a dimension-invariant module that enables knowledge sharing across heterogeneous architectures. To mimic the real-world task diversity, we propose a multi-modal PFL benchmark spanning 40 distinct tasks with distribution shifts over time. Extensive experiments shows that our proposed method significantly outperforms the state-of-the-art PFL methods under heterogeneous scenarios.
Executive Impact: Unlocking Personalized AI at Scale
This research introduces FedMosaic, a groundbreaking framework addressing the critical challenges of data and model heterogeneity in Personalized Federated Learning (PFL). By enabling efficient knowledge sharing across diverse client models and tasks, FedMosaic paves the way for truly personalized, privacy-preserving AI.
0
Avg. Self-Performance Gain
0
Avg. Generalization Gain
0
Communication Cost Reduction
0
Diverse Tasks Supported
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
The Dual Challenge: Data and Model Heterogeneity in PFL
Traditional Personalized Federated Learning (PFL) methods often operate under simplified assumptions: clients using identical models and handling homogeneous data. In real-world enterprise scenarios, especially with the rise of agentic AI, this is rarely the case. Clients frequently possess diverse computational resources leading to varied model architectures (model heterogeneity), and they engage with highly personalized tasks and continuously evolving data distributions (data heterogeneity). These factors hinder effective knowledge sharing and personalization.
Enterprise Process Flow: Heterogeneous PFL Challenges
Clients with Data Heterogeneity (Diverse Tasks, Distribution Shifts)
→
Clients with Model Heterogeneity (Different Architectures/Scales)
→
Local Model Training (Challenges in Sharing)
→
Server Aggregation (Infeasible with Mismatched Models)
→
Limited Personalized Global Model
FedMosaic: Bridging Heterogeneity with RELA and Co-LoRA
To overcome the limitations of traditional PFL, we introduce FedMosaic, a novel framework designed for realistic heterogeneous environments. FedMosaic comprises two key innovations:
- RELA (Relevance-guided Aggregation): A task-relevance-aware model aggregation strategy that minimizes parameter interference under diverse data distributions. It enables clients with related tasks to share knowledge more effectively by constructing customized global models based on task-relatedness.
- Co-LoRA (Collaborative-LoRA): A dimension-invariant module that facilitates seamless knowledge sharing across models with heterogeneous architectures and scales. This addresses the challenge where aggregating model weights becomes infeasible due to architectural mismatches.
Enterprise Process Flow: FedMosaic Framework
Client Local Training (Personalized Co-LoRA)
→
Upload Sanitized Gradients (Task Relevance Proxy)
→
Server: Measure Task Relevance (RELA)
→
Server: RELA-Guided Co-LoRA Aggregation
→
Distribute Customized Global Co-LoRA
→
Client Personalized Model Update (Local + Global)
DRAKE: A Comprehensive Multi-Modal PFL Benchmark
To rigorously evaluate FedMosaic and foster research in realistic PFL, we introduce DRAKE, a novel multi-modal PFL benchmark. Unlike prior works that simulate heterogeneity via simple data splits, DRAKE is designed to mirror real-world complexities:
- Task Heterogeneity: Each client is assigned a distinct multi-modal task (e.g., VQA, visual reasoning, visual relation), spanning 40 diverse tasks.
- Dynamic Distribution: Incorporates temporal distribution shifts, reflecting the evolving nature of real-world data.
- Generalizability Evaluation: Includes unseen tasks to test the model's ability to generalize to novel scenarios.
DRAKE represents the first multi-modal FL benchmark that considers both data heterogeneity and temporal distribution shifts, making it ideal for evaluating personalized AI systems.
| Benchmark | Multi-Data Sources | Distribution Shifts | Multi-Image Support | Multi-Modalities | Unseen Evaluation |
|---|---|---|---|---|---|
| FedDAT (AAAI 2024) | |||||
| PerAda (CVPR 2024) | |||||
| FedMultimodal (KDD 2023) | |||||
| DRAKE (Ours) |
Co-LoRA: Dimension-Invariant Knowledge Sharing
Co-LoRA is a core component of FedMosaic, enabling collaboration across heterogeneous model architectures. Unlike conventional LoRA, which uses dimension-dependent matrices A and B, Co-LoRA introduces dimension-invariant modules P and Q. These modules depend only on the low-rank size r, making them directly shareable across models of varying hidden dimensions and depths.
To maximize expressive capacity and ensure effective knowledge transfer, Co-LoRA employs a strategy of block-wise aggregation (aligning layers at similar relative depths based on CKA similarity, e.g., Llama-1B layer 8 strongly aligns with Llama-3B layer 14) and weight alignment (ensuring heterogeneous models share the same initialization). This significantly reduces aggregation error and enhances performance.
Enterprise Process Flow: Blockwise Co-LoRA
Input (Pre-trained Weight)
→
Block 1 (Co-LoRA Applied to Final Layer)
→
Conventional LoRA Layers (Within Block)
→
... (Intermediate Blocks) ...
→
Block NB (Co-LoRA Applied to Final Layer)
→
Output (Personalized Model)
Co-LoRA Capacity Theorem
r2 Dimension for Maximum Weight Update Space CapacityTheorem 1 states that with orthogonal initialization of matrices A and B, the span of Co-LoRA's weight update space reaches a maximum of r2 dimensions, optimizing its expressive power.
RELA: Task-Relevance-Aware Aggregation with Privacy
RELA (Relevance-guided Aggregation) addresses data heterogeneity by allowing clients to share knowledge selectively based on task relatedness. Instead of naive model averaging, RELA constructs a customized global model for each client, prioritizing contributions from clients performing similar tasks.
This is achieved by measuring task relevance through sanitized client-wise gradients, derived from a small-scale frozen pre-trained model. To ensure privacy and reduce transmission costs, these gradients undergo exponential moving average (EMA) updates, Gaussian noise addition, and gradient compression (randomly selecting 40% of dimensions). This process effectively mitigates privacy risks from gradient inversion attacks while preserving crucial task similarity information.
| Defense Method | Rouge-L ↓ (Lower is Better) |
|---|---|
| Full layer gradient | 0.2952 |
| Clipping + Noise (Bietti et al., 2022) | 0.2720 |
| SVD Truncation (Luo et al., 2025) | 0.2688 |
| Last layer gradient + EMA aggregated + Noise + Compression (Sanitized Gradient) | 0.0653 |
Communication Efficiency
10.9% Lower Communication CostFedMosaic achieves significant communication cost reduction compared to even the most efficient baselines, ensuring scalability in real-world deployments.
Superior Personalization and Generalization
Extensive experiments on the new DRAKE benchmark and existing PFL benchmarks (HFLB, Fed-LLM-Large) demonstrate that FedMosaic significantly outperforms state-of-the-art PFL methods across diverse heterogeneous scenarios.
FedMosaic consistently achieves higher personalization ('Self' accuracy) and generalization ('Others' accuracy), even with varying model architectures (Llama-based, Qwen-based MLLMs) and scales (1B vs. 3B models). This is attributed to RELA's effective task-relevance modeling and Co-LoRA's seamless cross-architecture knowledge transfer.
| Method | Self (Alast ↑) | Others (Alast ↑) |
|---|---|---|
| SFT | 65.79±0.20 | 47.66±0.13 |
| DITTO (ICML 2021) | 59.91±0.18 | 47.45±0.36 |
| FedDAT (AAAI 2024) | 58.47±1.10 | 48.91±0.42 |
| PerAda (CVPR 2024) | 59.75±1.06 | 47.30±0.45 |
| FedMosaic (Ours) | 67.86±0.51 | 51.16±0.04 |
Case Study: Client 9 Performance Boost
For Client 9, utilizing a LLaVA-Llama3.1-8B model, FedMosaic delivered a 6.5% increase in Self-Alast accuracy (from 73.65 to 78.41) and a 9.4% increase in Self-AAUC accuracy compared to Supervised Fine-Tuning (SFT). This highlights FedMosaic's capability to significantly enhance personalization for clients with larger, more complex models through effective knowledge sharing across heterogeneous architectures.
Calculate Your Potential AI ROI
Estimate the efficiency gains and cost savings for your enterprise by implementing advanced AI personalization with FedMosaic.
Estimated Annual Savings
$0
Annual Hours Reclaimed
0
Your Personalized AI Implementation Roadmap
A phased approach to integrating FedMosaic into your enterprise, ensuring a smooth transition and maximum impact.
Phase 01: Strategic Assessment & Data Readiness
Evaluate existing infrastructure, data heterogeneity, and privacy requirements. Prepare multi-modal datasets for federated learning, ensuring data quality and accessibility across diverse client environments.
Phase 02: FedMosaic Framework Integration
Integrate FedMosaic with Co-LoRA for heterogeneous model support and RELA for task-relevance-aware aggregation. Deploy the DRAKE benchmark to validate performance under realistic conditions and fine-tune hyperparameters.
Phase 03: Pilot Deployment & Performance Optimization
Initiate a pilot project with a subset of clients and tasks. Monitor personalization ('Self') and generalization ('Others') metrics. Optimize communication costs and adaptation speed for your specific enterprise environment.
Phase 04: Scaled Rollout & Continuous Improvement
Expand FedMosaic deployment across your client ecosystem. Establish continuous learning loops and feedback mechanisms to ensure ongoing personalization and adaptation to evolving tasks and data distributions.
Ready to Transform Your Enterprise with Personalized AI?
Book a complimentary strategy session with our AI experts to explore how FedMosaic can address your unique challenges and drive innovation.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat