In this research, a dataset has been crawled from an e-commerce site in Pakistan containing 7710 reviews written in Roman Urdu. Here we have each review containing a rating and a label i.e. OR (Human-generated original review) and F (Fake computer-generated review), category of product, and the text of the review. To achieve better results and find the outperforming model, the approach consists of four phases; review collection, pre-processing, training classifier, and testing reviews to be classified. Figure 1 illustrates the proposed framework.

In our methodology, we employed a diverse set of nine machine learning algorithms to comprehensively analyze their effectiveness in detecting fake reviews. These algorithms include Support Vector Machine, Naïve Bayes, Logistic Regression, Decision Tree, K-Nearest Neighbors, Random Forest, Recurrent Neural Network (RNN), Convolutional Neural Networks (CNN), and Long Short-Term Memory (LSTM). The reason behind this selection was to ensure a robust evaluation, considering the varied strengths and capabilities of each algorithm. Naïve Bayes is known for its simplicity and efficiency, while Support Vector Machine excels in handling complex data. Logistic Regression provides a probabilistic approach, and Decision Tree and Random Forest offer ensemble learning. K-Nearest Neighbors utilizes proximity-based classification. The inclusion of deep learning models such as RNN, CNN, and LSTM allows for a nuanced exploration of sequential patterns in reviews. This comprehensive array of algorithms enables a thorough comparison, ensuring a nuanced understanding of their performance in the context of fake review detection.

Dataset Collection:

We collected a total number of 7710 reviews from Daraz. pk which is well well-known e-commerce site, widely used in Pakistan. The reviews that we collected contain Roman Urdu. Each review contains, the reviewer’s username, date, and the category of the product. We have utilized 30% of the dataset for training purposes, while the remaining is reserved for testing. Out of these reviews, we have labeled 1710 reviews as fake out of which, 710 were deliberately written by us. The rest of the reviews are considered real.

Preprocessing:

The first phase in this approach starts with the preprocessing of the dataset [22] an essential step in the approach of machine learning. The step of data processing is so important because of the roughness of a real-world dataset like the usage of stop words, commas, numbers, etc. Sequential steps of preprocessing data have been applied to make raw data from the crawled dataset for several activities involving computation. The steps are given as:

Tokenization:

One of the frequently encountered common and important techniques in natural language processing is tokenization. It is commonly regarded as a fundamental step before applying any processing technique. The words in the text review are divided separately into “tokens”. For example, if the review is “The product is amazing”, after tokenization it will appear as (“The”, “Product”, “is”, “amazing”) [24].

Cleaning of Stop Words:

Stop words [23] are supposed to hold no value in the process of fake review detection. Some typical instances of stop words include; a, an, this, the, or. We cleaned all stop words to reduce the unnecessary word count and to make the process of fake review detection smooth.

Removal of Non-Alphabetic Words:

This step is used for removing punctuation marks, full stops, and digits from the reviews. The non-alphabetic words have no major impact on the processing of raw data.

Feature Extraction:

The unnecessary attributes can cause reduced accuracy of the model [25]. Therefore, feature extraction is performed to improve the efficiency of pattern recognition in machine learning systems. It is mainly used in the reduction phase of the dataset, which molds it into useful features that are fed into deep learning and machine learning models. Afterward, the data appears to be more meaningful.

Figure 1: Proposed framework diagram

Several approaches in the literature have been developed to extract features from reviews. One of the popular approaches is textual features [26]. The textual feature contains sentimental classification [5] which depends upon negative and positive stances in the review e.g. “weak”, “good” or “strong”. Sometimes, the similarity of Cosine is used as a feature that refers to the cosine of the angle formed between two n-dimensional vectors and the result obtained by dividing the dot product of two vectors by the product of their magnitudes [27]. Another feature is true-false identification, also referred to as TF-IDF. This method counts the frequency of TF (True and false) and IDF (inverse document) scores for all words. The product of IDF and TF scores of the term is referred to as the weight of the TF-IDF term [28]. A confusion matrix is also used for the classification of reviews across four results; FP (Fake Positive), TP (True Positive), FN (Fake Negative), and TN (True Negative).

Equivalently, other features are behavioral and user personal profiles. These features can be used to identify spammers based on frequent time stamps or if the user is using redundant reviews and posting them everywhere regardless of the domain of the product. In this work, we used BoW (Bag of word features), also referred to as n-gram features, with TF-IDF. BoW features represent text into numerical representation i.e. the vector of numerical features that can be used to train the models. The features represent the frequency of each word appearing in the whole dataset of reviews. TF-IDF on the other hand is used to score each instance in the review text to make the detection more accurate.

Feature Engineering:

In this research, we combined TF-IDF with BoW to extract more accurate features from the vocabulary built by BoW and used these features to train multiple classifiers. In our crawled dataset, we have 7710 reviews and 7486 total vocabularies. TF calculates how often a word occurs across the document and the IDF is the measure of how important the word is. With the use of IDF, the unnecessary words in our dataset with 0 scores were reduced and we obtained the words with higher scores. A high score refers to more importance. Finally, as the product of IDF and TF, we obtained the TF-IDF score of the BoW. These features as a product of TF and IDF on BoW are taken into consideration to identify fraudulent reviews and conduct performance analysis on machine learning models.

Model Evaluation and Hyperparameter Tuning:

In this phase, following the preprocessing of our dataset through techniques e.g. the Bag-of-Words (BoW) and The Term Frequency-Inverse Document Frequency (TF-IDF), we subjected the refined dataset to a rigorous evaluation using nine distinct machine learning models. These models, including Naïve Bayes, Support Vector Machine, Logistic Regression, Random Forest, Decision Tree, K-Nearest Neighbour, Recurrent Neural Network (RNN), Convolutional Neural Networks (CNN), and Long Short-Term Memory (LSTM), were chosen to comprehensively assess their efficacy in fake review detection. To optimize their performance, we systematically adjusted their hyperparameters, seeking the configurations that would yield the highest accuracy in our specific context. This meticulous process ensures a robust evaluation of each model's capabilities, facilitating an informed comparison of their effectiveness in identifying counterfeit reviews.

Naïve Bayes Machine Learning Probabilistic Algorithm:

Our initial trial involved implementing the Naïve Bayes machine learning probabilistic algorithm, which relies on principles of the Naïve Bayes theorem and is used widely in classification tasks. We have tuned the hyperparameter alpha to 0.7 to handle the problem of zero frequency. Tuning this hyperparameter greatly improved the results and we found an f1 score of 0.73.

The Decision Tree Classifier:

The tree-like structure efficiently portrays a collection of decisions and their potential outcomes, making it effective for resolving decision-based challenges. In our situation, we have chosen a depth of three strikes which is a good balance between model complexity and accuracy and is not too shallow or too deep. We observed that a decision tree with a shallow depth is more appropriate for smaller datasets and prior knowledge also suggests that a depth of three is sufficient to capture the relevant patterns and relationships in the data. It obtained an f1 score of 72.

Random Forest Classifier:

After the decision tree, we tested by increasing the number of trees, which is the case in a random forest classifier. We set the number of trees to 80 which is an approximately optimal value for our dataset. We also observed that the accuracy at 100 and 80 trees was the same, but the training time was reduced by 0.51 when we selected 80 trees. The model produces the f1-score of 0.75.

K-Nearest Neighbour:

The algorithm being described is a lazy learning and non-parametric approach that uses a simple distance metric to find the K-nearest neighbors to a given data point. We tried several K values for cross-validation and found that setting K = 3 balances the trade-off between bias and variance. This value made the model more flexible with low bias and high variance. With k=3, it obtained an f1 score of 0.75.

Support Vector Machine (SVM):

It finds the best possible decision boundary (or hyperplane) which is used to separate the data points among different classes. SVM can also handle dimensional data and is effective in dealing with nonlinear classification problems by using regularization parameters (C), and kernel tricks. We tuned C to 0.6, which is a good balance between bias and variance, and used Polynomial Kernel as a kernel trick. Tuning these hyperparameters got us an f1-score of 0.82.

Logistic Regression:

Logistic regression aims to predict a categorical outcome variable that is based on a minimum of one input variable or feature. We have L2 as the penalty and made 100 iterations for the solver. After tuning the value of C to 1.0, the model produces 0.77 as the f1 score.

Neural Networks:

In this phase, we examined three deep neural networks to find the most outperforming model. Deep learning methods, such as most representative neural networks, outperform traditional machine learning models in efficiently extracting valuable data features. The dataset undergoes testing using the following deep-learning models:

Convolution Neural Networks (CNN):

In the natural language processing task, CNN plays a momentous role in pulling in local features for classification. We have used 3 convolution layers with 32, 3 x 3 size filters, and a stride of 1. By tuning the hyperparameters to these given values and using ReLU (Rectified Linear Unit), which is an activation function, the model produces an f1-score of 0.75 on our dataset.

Recurrent Neural Network (RNN):

In comparison to conventional feed-forward neural networks, which process the data in one pass, RNNs can also keep a state or memory of past inputs, which makes them well-suited for sequential data. We used 32 hidden layers, a learning rate of 0.001, and tanh (hyperbolic tangent) as an activation function. The model obtained an f1-score of 0.72.

Long-short-term memory (LSTM):

It is an advancement of the RNN model. We used this model because of the long-term dependency problem in RNN. LSTM networks are designed to handle long-term dependencies and are capable of learning and remembering information over long periods. We have used 32 LSTM layers, 64 batch sizes, and a hyperbolic tangent (tanh) function as an activation function that produces an f1-score of 0.77, which is better than RNN.

Results and Performance Analysis:

We used six traditional machine learning algorithms; Support Vector Classifier, Logistic Regression, Decision Tree Classifier, K Nearest Neighbors, Multinomial Naive Bayes, Random Forests Classifier, and three neural networks; CNN, RNN, and LSTM, on our dataset. Two predefined labels are given to our dataset; F (Fake) which refers to fake reviews and OR (Original) which is real and written by a human. We have divided the dataset into 1710 F and 6000 OR reviews. Each review instance in the dataset contains a rating, Label (F or OR), and category of the product. Table 1 summarizes the statistics of the dataset.

Table 1: Statistics of the dataset

Before passing the dataset through the algorithms, we applied several text-pre-processing techniques to the dataset to make it free from roughness. After pre-processing, the code produces a new pre-processed XML file which is further utilized in the experiments. Counter Vectorizer takes place in the next step which converts the text to a number in terms of the sparse matrix. We then made BoW for our document to be used for further feature extraction from the dataset. To get better features, we further used TF-IDF with BoW. From TF-IDF we obtained the score of each instance in the review and it greatly improved the results.

We trained the classifiers using 30% of our dataset and 70% for testing. After training, we tested it with nine machine-learning models and generated the classification report for each which is further discussed in the results section.