AI / Deep Learning for Edge Computing

BN-Free Design with WS-Conv

The core innovation of WSFracBNN lies in the strategic removal of Batch Normalization layers. While BN is critical for stabilizing training and improving accuracy in traditional NNs, it introduces significant floating-point computational overhead. To counteract the severe accuracy drop from BN removal, the model integrates Scaled Weight Standardization Convolution (WS-Conv). WS-Conv normalizes convolution kernel weights, acting as a regularization technique that stabilizes training without the computational burden of BN.

Additionally, drawing inspiration from ReActNet, the model incorporates RPReLU (Rectified Parametric ReLU) to dynamically adjust activation distributions, ensuring that 1-bit binarization captures essential features more effectively rather than introducing noise.

Advanced Optimization Strategies

To ensure robust training and optimal performance for the BN-free WSFracBNN, several advanced optimization techniques are employed:

• Adaptive Gradient Clipping (AGC): This method prevents exploding gradients, a common issue in BNN training due to the rugged loss landscapes. AGC adjusts clipping thresholds based on the norm ratio of gradients to kernel weights, enhancing training stability.
• Knowledge Distillation (KD): A teacher-student learning approach where a smaller student model (WSFracBNN) learns from a more complex, high-performing teacher model (NFNet-F0). KD helps the binary model achieve accuracy closer to full-precision networks by mimicking the teacher's output distribution.
• Adam Optimizer: Empirical analysis demonstrates that Adam outperforms SGD for BNNs. Adam's adaptive learning rates and momentum-based updates navigate the highly non-convex and rugged loss surfaces of quantized networks more effectively, mitigating issues like activation saturation and vanishing gradients.

Quantifiable Performance Gains

The WSFracBNN model, despite its significant architectural changes for edge deployment, achieves superior or competitive performance:

• Accuracy: Achieves 59.5% Top-1 accuracy on CIFAR-100, surpassing the baseline FracBNN by 0.6% and outperforming BiRealNet-18 and ReActNet-A by substantial margins.
• Computational Efficiency: Reduces Floating-Point Operations (FLOPs) to a near-zero 0.06 × 10^6 (from 84.3 × 10^6), while maintaining binary operations (BOPs). This results in a 54% reduction in total operations (OPs) compared to the baseline FracBNN.
• CPU Throughput: Demonstrates improved processing speed on CPU, confirming its practicality for real-world embedded applications.

These results validate the model's suitability for high-performance image recognition on resource-limited edge devices, offering an optimal balance between accuracy and computational cost.