Sentiment Trend Forecasting in E-Commerce Reviews Using Transformer-Based Representations and Time-Series Modeling

Abstract

The rapid growth of e-commerce platforms has generated large-scale user-generated textual data, creating opportunities for modeling not only static sentiment polarity but also the temporal evolution of consumer opinion. This study formalizes Sentiment Trend Forecasting (STF) as a predictive time-series problem in which contextual sentiment representations extracted from transformer models are aggregated into temporal signals and used to forecast future sentiment trajectories. The dataset consists of Amazon product reviews spanning 2003–2012, resulting in more than 450 weekly observations after temporal aggregation. Aggregated contextual sentiment signals are constructed from review-level embeddings generated using pre-trained BERT and RoBERTa models across discrete time intervals. Weekly sentiment series are modeled using ARIMA and Long Short-Term Memory (LSTM) architectures under a rolling forecasting protocol with a 52-week hold-out horizon. Forecasting performance is evaluated using Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE). Experimental results show that BERT-based sentiment signals achieve MAE = 0.00041 and RMSE = 0.00052, outperforming VADER (MAE = 0.084, RMSE = 0.101) and rating-based baselines (MAE = 0.205, RMSE = 0.254). Although RoBERTa-based signals yield low error values (MAE = 0.00012, RMSE = 0.00015), their near-constant output results in weak correlation with ratings (r = 0.15), limiting their interpretability. Statistical validation includes stationarity testing (ADF), residual diagnostics (Ljung–Box), and Diebold–Mariano tests. The Diebold–Mariano test confirms the statistical superiority of BERT-based forecasts (p < 0.01). The results confirm that contextual embedding-based sentiment representations provide predictable temporal signals for proactive monitoring of consumer opinion.