AI RESEARCH PAPER ANALYSIS

Advancing Speaker Erasure: From TGP to EGP+Triplet

The research explores several methods for achieving targeted speaker poisoning:

• Naive Baselines: Initial attempts involved pre-processing strategies like "Pretrained + Speaker Filtering" and "Pretrained + Ground Truth Filtering." However, these external filtering methods are vulnerable if model weights are public, as adversaries can bypass the pipeline.
• Teacher-Guided Poisoning (TGP): Building on the knowledge distillation paradigm, TGP adapts a teacher model to guide the student model in generating random speaker identities from the retain set when prompted with a forget set speaker.
• Encoder-Guided Poisoning (EGP): EGP refines TGP by directly using the style encoder's output as the fine-tuning target, bypassing teacher-generated outputs. This provides a cleaner optimization signal, leading to superior performance.
• Contrastive Learning (Triplet Loss): To further enhance privacy, a triplet loss objective is incorporated. This loss explicitly penalizes embeddings that remain similar to the forget set, pushing generated outputs away from negative references sampled from the forget set.

Robust Evaluation: AUC and FSSIM

To rigorously assess the effectiveness of the proposed methods, a comprehensive evaluation framework was developed, focusing on both utility and privacy.

Utility Metrics:

• Word Error Rate (WER): Measures speech intelligibility.
• UTMOS: An automated proxy for Mean Opinion Score, assessing naturalness.
• Speaker Similarity (SSIM): Cosine similarity between reference and synthesized speech embeddings, measuring identity preservation for the retain set.

Privacy Metrics:

• Area Under the Curve (AUC): Quantifies the separability of similarity distributions between retain and forget sets, providing a distribution-level assessment of prompt-output dissimilarity. An AUC of 1.0 indicates perfect separation.
• Forget Set Similarity (FSSIM): A stronger privacy metric that measures similarity between each generated sample and every speaker in the forget set. It includes Avg-FSSIM and Max-FSSIM, with the latter addressing worst-case leakage.

This framework ensures a nuanced understanding of privacy-utility trade-offs, moving beyond raw similarity scores to comprehensive, distribution-aware metrics.

Performance Highlights & Scalability Challenges

The research presents detailed results across settings with 1, 15, and 100 forgotten speakers. Key findings include:

• Effective Single-Speaker Erasure: EGP+Triplet achieves an AUC of 0.95 for a single forgotten speaker, demonstrating strong privacy without significant utility degradation.
• Scalability to 15 Speakers: EGP+Triplet maintains good performance for up to 15 speakers, showing a measurable similarity gap between retain and forget sets.
• The 100-Speaker Challenge: At 100 forgotten speakers, the distinction largely collapses due to increased identity overlap. Max-FSSIM remains high (around 0.91), indicating persistent worst-case leakage and the diminishing effectiveness of triplet loss.
• EGP Outperforms TGP: EGP consistently shows better performance, attributed to a cleaner optimization signal by directly targeting original style encoder representations, avoiding generative noise introduced by a teacher model.

These results highlight the potential of parameter-modifying approaches for targeted speaker erasure but also underscore the significant challenges in scaling these methods to a large number of identities due to latent space crowding.