Natural Language Processing

Commerce User Review Sentiment Analysis Model Based on BERT and Transformer

Sentiment analysis of user reviews plays a crucial role in e-commerce platforms for understanding customer satisfaction, monitoring product reputation, and improving service quality. Traditional sentiment analysis methods, including lexicon-based approaches and conventional machine learning algorithms, often struggle with capturing contextual semantics, handling domain-specific expressions, and addressing the linguistic complexity inherent in user-generated content such as informal language, sarcasm, and mixed sentiments. This paper proposes a novel sentiment analysis model that integrates BERT (Bidirectional Encoder Representations from Transformers) for contextual text encoding with an enhanced Transformer architecture specifically designed for fine-grained sentiment classification in e-commerce reviews. Our approach incorporates four key innovations: (1) domain-adaptive pre-training on 2 million unlabeled e-commerce reviews to capture domain-specific vocabulary and linguistic patterns, (2) sentiment-aware attention mechanisms with learnable query vectors that explicitly identify and emphasize sentiment-bearing expressions such as opinion words, negations, and intensifiers, (3) multi-level feature aggregation that combines token-level representations from different BERT layers through hierarchical attention to capture both syntactic and semantic information, and (4) focal loss optimization to effectively handle the class imbalance problem commonly observed in real-world review datasets. Our model leverages BERT's pre-trained bidirectional language representations to capture deep semantic features and contextual dependencies from reviews, while the enhanced Transformer layers with sentiment-aware attention enable effective modeling of long-range dependencies and complex sentiment-bearing patterns across different linguistic granularities.

Authored by Tuli Chen in ICADI 2025 (November 28-30, 2025)