Hierarchical Windowed Graph Attention Transformer Encoder and a Large Scale Dataset for Indian Sign Language Recognition

Sign Language (SL) recognition is a crucial task in computer vision research to identify signs from the articulation of hand shapes, hand gestures, body movements and facial expressions. Building robust SL recognition systems requires large-scale datasets with accurate labels. However, such datasets are scarce for Indian Sign Language (ISL). This paper addresses this gap by introducing a large-scale dataset of isolated ISL signs. The dataset encompasses 2002 frequently used words in the deaf community, signed by 20 deaf adult signers and contains 40033 isolated videos. We split the dataset into train, validation and test sets with no signer intersection for benchmarking models. We follow the keypoint-based approach and report the results of some existing models on this dataset. Traditional architectures process spatial and temporal features sequentially. We introduce a novel keypoint-based model called the Hierarchical Windowed Graph Attention Transformer Encoder (HWGATE) that simultaneously aggregates spatial and temporal features using a unified spatio-temporal graph. Additionally, it enhances robustness through keypoint grouping. This model achieves an accuracy of 95.67% on the presented dataset. We pre-trained the proposed model on the new dataset and then fine-tune it on the smaller ISL INCLUDE dataset, which yields state-of-the-art results of 98.28% accuracy, beating its baseline by 12.68 percentage points.