Audio & Speech Processing

The proposed architecture builds upon established self-supervised speech models with targeted modifications to enable noise awareness. The foundation for this work is the wav2vec2-XLSR-53 model, which provides multilingual speech representations through cross-lingual pretraining. The core innovation involves adding a parallel classification pathway to the existing transcription decoder. This noise detection head consists of a linear transformation layer followed by softmax activation, producing probability distributions over noise versus speech categories. The architectural modification enables the model to learn representations useful for both transcription and noise discrimination simultaneously.

Training optimization combines two objective functions: connectionist temporal classification loss for transcription and cross-entropy loss for noise classification. The total training objective is computed as a weighted combination of these losses, where the relative weighting can be fixed or learned during training. This adaptive loss weighting parameter enables dynamic balance between transcription and classification objectives. Experiments explored alternative feature combination approaches, including positional encoding from the convolutional feature extractor combined with contextual representations from transformer layers, creating richer feature representations for both decoding pathways.