Equipment and Setup:

Two cameras are used to record the complete dataset. Camera 1 is a versatile featureset EOS M50 digital camera from CANON, and Camera 2 is EOS 700D digital camera from CANON. Three Newer LED 500 Ultra High-Power dimmable lights are used; one of them is used to illuminate the subject and two others for background radiance and shadow elimination. Throughout the dataset creation, the lighting remained cold and with stable color temperature, and as a backdrop white project or a white background wall was consistent.

High-definition 4K videos are recorded using Canon M50 and Canon 700D, with a 1920 x 1086 at 25 frames per second for Canon M50 and 59.94 frames per second using Canon 700D, saved in .MP4 file format. Tab. 1 expands the details of the equipment used. The first thirty people are recorded with Canon M50, records while the remaining seventy people are recorded with Canon 700D. Overall, the set of videos includes 34,650 frames recorded for seven classes of dynamic gestures. The size of videos for every gesture varied from 1 second to 3 seconds due to the speed of every individual performing any motion or action and the gesture’s versatility.

Table 1: Details of equipment and videos for dynamic signs

Objective and Procedure:

The PkSLMNM dataset was collected following an approved protocol by the "Ethical Review Committee (ERC) of the School of Engineering and Applied Sciences at Bahria University Karachi Campus, under the application ERC/ES/005". This thorough ethical oversight guaranteed that all ethical principles and guidelines were followed during the data collection method. All 100 people who helped to create the dataset, gave their informed agreement to be a part of this study, underscoring the importance of ethical data collection and use.

In the interest of ethical research procedures, it is critical to state that the PkSLMNM dataset was obtained solely for research purposes and that the data-collecting technique was not damaging to the participants in any manner. Each participant was filmed for seven dynamic videos, with each recording session lasting no more than 15 minutes. The dataset includes a broad sample of male and female individuals ranging in age from 20 to 50 years old, reflecting the adult population.

Participants were taught to keep the desired facial expression for a brief period while being recorded while facing forward to their body position. During data collection, strict quality control methods were implemented, including real-time image review to ensure participants did not move, blink, or display unintentional expressions. These reviews also confirmed the availability and quality of all relevant photographs. Notably, the dataset retains realism by not regulating the subjects' apparel (shirts), allowing for slight variations in the individual’s position and facial states. All other areas of data collecting, on the other hand, were thoroughly managed to assure data quality and consistency.

Dynamic gestures offered distinct obstacles during data collection, with individuals moving at variable speeds, resulting in video lengths spanning from 1 to 3 seconds. Certain elements also contributed to data robustness. Occlusion induced by persons wearing glasses, male subjects with beards, and certain female participants wearing a headscarf are among these issues. Furthermore, the backdrops in the sample vary significantly due to variances in apparel color and style. Furthermore, the dataset includes a wide range of skin tones and facial features, which adds to the complexity of algorithm evaluation in the context of sign language detection.

Figure 1: Glimpses of the developed dataset

Lexicons of Dataset:

The current PkSLMNM dataset is a comprehensive collection of both manual (handbased) and non-manual (facial and body-based) gestures. As depicted in Figure 1, the dataset encompasses a set of seven adjectives—bad, best, sad, glad, scared, stiff, and surprise—each of which portrays one of seven primary expressions: disgust, neutral, happiness, sadness, fear, anger, and surprise, respectively. All the recorded video clips have an average duration of 2 seconds, and the variations in input clip sizes, ranging from 3MB to 15MB, can be attributed to the utilization of two different types of cameras, variations in individual speed, and the versatility of the gestures [12].

Facial expressions are complex, involving the activation of minute muscle movements in the face, known as micromotor muscles. These muscular movements are critical in determining an individual's mood or feeling. Facial Action Coding System (FACS), also known as Facial Action Units (FAUs), was a way to categorize an individual's emotional state in 1978. This technique takes into account several face units such as the forehead, brows, eyelids, nose, cheeks, and lips. Table 2 contains a full explanation of the facial action units and associated hand movements, as well as minor body movements, that correspond to all of the gestures or signs employed in Pakistan Sign Language (PSL), which form the basis of our research. The current dataset is visually reviewed and collected in the presence of domain experts, data is also cleaned up, and all erroneous signs are deleted. Balanced distribution among classes with a proper labeling scheme is followed.

Methodology:

During the first phase of our research, we carefully collected an independent dataset to address the lack of understanding of manual and non-manual gestures in Pakistan Sign Language (PSL). For the validation of this newly produced dataset, we used the YOLO-Face architecture, which incorporates face identification as well as exact Region of Interest (ROI) selection and cropping. Figure 2 shows the flow of methodology.

This procedure has verified that our dataset was accurate and reliable. Following that, we classified seven fundamental expressions using Transfer Learning, leveraging prior knowledge to obtain a high level of accuracy in expression classification. Figure 2 depicts our methodology's sequential procedure, demonstrating the systematic advancement from dataset collection to face detection, ROI selection, and the final stage of expression categorization.

YOLO-Face Architecture (Face ROI Selection):

The method for Face ROI recognition, as depicted in Figure 3 comprises four distinct stages: face detection, data preprocessing, feature extraction, and face ROI cropping.

• The first stage, face detection, is in charge of recognizing and finding faces within the input image or video frame.

• The second stage involves data preprocessing, in which image processing techniques are used to prepare the discovered images for subsequent processing by machine learning models.

• The third stage focuses on feature extraction, in which significant features from the prepared photos are retrieved, allowing for a thorough depiction of the facial traits.

• ROIs are detected and cropped in the final phase, face recognition, by comparing the features retrieved from the input data based on their facial qualities.

Face Detection:

The section delves into the initial step of our methodology, which is devoted to detecting faces in input data even in spontaneous and dynamic circumstances. Face detection at this level requires the ability to deal with a slew of nonlinear elements in the input images, such as fluctuations in luminance, facial posture, and facial magnitude. The face detection method was chosen after a thorough study of numerous approaches appropriate for real-time applications, such as YOLO-Face.

Figure 2: Workflow of the Research

The proposed method uses a one-phase deep learning module, as opposed to a twophase procedure such as Faster-RCNN [13]. YOLO, or "You Only Look Once," has the unusual ability to forecast classes connected with respective bounding boxes at the same time. One of the key benefits of the one-phase strategy is its incredible speed, which outperforms two-phase systems. This speed and efficiency are especially beneficial in real-time circumstances, making it an excellent choice for our face identification phase.

Figure 3: Real time face ROI extraction workflow

YOLO Face:

The importance of machine learning and deep learning cannot be neglected in this era specifically for image processing [17]-[33]. The section explains the primary characteristics of the proposed model that utilizes YOLOv3 [13] as the detection model, which depends on object recognition methods using deep learning. Following this mechanism, face detection modules perform significantly better.

Prediction of Classes. YOLO-Face works on binary classification which means it only recognizes two classes; face or no face, instead of multiclass. The proposed system is based on an object detection model, which reduces the error at the output of the model using a multipart loss function. There are four parts of this loss function which include loss of classification, regression, and confidence in anchors with and without objects. These four parts of loss are divided with a 2:1:0.5:0.5 ratio.

Network for extracting features. Features are extracted in YOLO-Face using the darknet-53 network [14], it is a concatenation of two other feature-extracting networks; darknet-19 of YOLO-V2 and residual network of newfangled with 53 layers. When the feature extractor of the proposed network is compared with the darknet-19 of YOLO-v2. The first network outperforms the latter. In terms of efficiency, it significantly worked better when compared to ResNet-101 and ResNet-152. The performance of YOLO-v3 degrades while detecting small-scale objects in YOLO-v3, which is the only main problem. Hence, the proposed network incremented the number of layers of the actual darknet-53 network in the two initial residual blocks. These amendments in the structure of the network are made before the characteristics of small objects reduce more on the feature map resulting in an increase in the quality of the features of the face. The number of layers in the initial two blocks has been incremented to 4x and 8x from 1x and 2x. The feature map of the network is scaled down with stride 2 of the convolutional layer which is considerably bigger compared to YOLO-v3. The proposed network uses deeper Dark Net which is a set of 71 convolutional layers. Figure 4 depicts the entire network structure. YOLO-Face has an equivalent recognition speed as Face Attention Network (FAN) according to [15] results, however, FAN is more inefficient.

Figure 4: YOLO-Face architecture is adapted to find face ROIs in our dynamic PkSLMNM dataset

Prediction of Various Scales:

YOLO-Face uses a notion equivalent to FPN [16] as shown in Fig. 3, it detects different faces and predicts their respective bounding boxes around three various scales. Using multiple sizing feature maps, the approach outperforms multi-scale detection. As the feature map generator, the outcome of the last residual layer is chosen that is embellished to create a pyramid structure. The detection mechanism is achieved when combined with the two preceding blocks which have not been sampled by 2x yet. The anchor boxes of the object recognition systems are modified to make them competent for face detection, which implies that anchor boxes are created narrow and tall rather than with heights less than width.

Face classification and identification models have been improving in the past few years by modifying structure and optimizing the loss function, such networks include VGG-Face, Face Net, Cosface as well as Ring Loss. For modeling the multiclass classification tasks, Softmax with cross-entropy loss function has drawn a lot of attention and can be measured by equation 1:

Here total number of identified samples is represented by N, one-hot encoded expression of the ith sample is labeled as yiЄ{0,1}j, with Softmax and ground truth class the predicted probabilities are represented by ŷig.

The study aims to improve the precision of the classification results of the Face Net through L2 normalization and softmax with cross entropy as the loss function. The modification in the loss function has made CNN learn features between various classes more distinguishably, subsequently getting smaller and larger intra-class distances. When compared with the traditional methods, the proposed network outperformed work effectively better as shown in the results that in the LFW database, the final precision is enhanced.

Transfer Learning (VGG 16 Architecture):

Transfer Learning with the VGG16 architecture is a powerful deep-learning approach, especially when dealing with image recognition or classification applications. VGG16, a CNN architecture, is well-known for its depth and ability to capture intricate hierarchical structures inside images. Transfer Learning aims to capitalize on VGG16's knowledge gained by pretraining on a large and diverse dataset like ImageNet [11].

Model: "VGG16 Transfer Learning Model"

The above given architecture summary of the VGG16 model, including its layers and the number of parameters in each layer. The model consists of convolutional layers (Conv2D), max-pooling layers (MaxPooling2D), and fully connected layers (Dense). The "blockX_convY" layers represent convolutional layers within different blocks, and "blockX_pool" represents max-pooling layers. The final layers include a flattening layer followed by three fully connected layers ("fc1," "fc2," and "predictions"), with the last one having 1000 output nodes for ImageNet's 1000 classes. The model has a total of 138,357,544 parameters, all of which are trainable.