Efficient Methods of ASL Video Classification Using 3DCNNs

Title: Summarizes the main idea of your project.

Who: Names and logins of all your group members.

Anushka Narayanan - anaray34 Colden Bobowick - cbobowic Kaitlyn Williams - kwilli43

Check In 2

Introduction: What problem are you trying to solve and why?

Developing robust tools for sign language classification can help alleviate the communication gap between the hard-of-hearing community and those unfamiliar with ASL. Deep learning is an ideal approach to this problem as it is able to capture complex underlying patterns in high-dimensional data such as videos. An algorithmic approach of classifying ASL clips is infeasible, due to the significant variation between signers and the high complexity of the data. Though currently, models such as CNN-LSTMs, 3-D CNNs, hidden markov models etc. aim to tackle this problem, there does not seem to be a consistent approach. In this project, we aim to compare three convolutional architectures, Conv2+1D, Conv3D and Conv2D-LSTM models in an efficient manner by reducing the number of trainable parameters involved. The Conv2D-LSTM approach is a potential novel architecture used for sign language classification which may offer improved accuracy and insights given the spatial and temporal nature of sign language videos.

Related Work: Are you aware of any, or is there any prior work that you drew on to do your project? Please read and briefly summarize (no more than one paragraph) at least one paper/article/blog relevant to your topic beyond the paper you are re-implementing/novel idea you are researching. In this section, also include URLs to any public implementations you find of the paper you’re trying to implement. Please keep this as a “living list”–if you stumble across a new implementation later down the line, add it to this list.

The MS-ASL paper, MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language [1] explores baseline architectures such as to classify ASL videos to develop a comprehensive benchmark dataset for the community In Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset [2], the Two-Stream Architecture using 3-D Convolutional Networks for a Kinetic Video Dataset informs the architecture the MS-ASL paper utilizes as a benchmark model This paper VoxNet: A 3D Convolutional Neural Network for real-time object recognition [3] talks about how they used different types of cameras to process 3D data and how they tackled the analysis of this data using a 3D CNN. Instead of using many layers to implement their design, they decided to go with fewer convolutional layers to increase the efficiency of their model. We are wanting to use this idea of using fewer layers to increase efficiency, combined with increased image augmentation to increase accuracy, in our exploration of classifying ASL videos. The Conv2+1D Architecture is informed by A Closer Look at Spatiotemporal Convolutions for Action Recognition [4] which uses 3-D convolution layers but processes the spatial and temporal dimension separately. Public Implementation: https://www.tensorflow.org/tutorials/video/video_classification

Data: What data are you using (if any)? If you’re using a standard dataset (e.g. MNIST), you can just mention that briefly. Otherwise, say something more about where your data come from (especially if there’s anything interesting about how you will gather it).

How big is it? Will you need to do significant preprocessing? We are using the MS-ASL dataset, which is a 1000 class video dataset sourced by Microsoft in 2018. The dataset is in the form of a JSON file with links to Youtube videos, as well as bounding boxes, start and end times, and class names. Significant preprocessing will need to be done in order to obtain the cropped, clipped videos locally for training purposes. Although the dataset contains over 24 hours of videos, a significant portion of the videos provided are private or unavailable. Thus, we will be using a subset of ~60% of the MS-ASL dataset.

Methodology: What is the architecture of your model?

How are you training the model? If you are implementing an existing paper, detail what you think will be the hardest part about implementing the model here. If you are doing something new, justify your design. Also note some backup ideas you may have to experiment with if you run into issues. The Conv2+1D model [4] proposes splitting 3-D convolution across temporal and spatial dimensions increases accuracy over 2-D convolutions The Conv 3-D model is an existing baseline models used in MS-ASL that has shown to be accurate in classifying sign-language videos [1] [2] The Conv 2-D LSTM model will different compared to the standard CNN-LSTM (2-d convolution layer with an lstm layer). A Conv2d-LSTM is an LSTM layer where the internal matrix multiplication operations are replaced with convolution operations with less trainable parameters. We propose to use simple architectures that reduce the number of trainable parameters proposed by [3] to produce efficient models

Metrics: What constitutes “success?”

What experiments do you plan to run? For most of our assignments, we have looked at the accuracy of the model. Does the notion of “accuracy” apply for your project, or is some other metric more appropriate? If you are implementing an existing project, detail what the authors of that paper were hoping to find and how they quantified the results of their model. If you are doing something new, explain how you will assess your model’s performance. What are your base, target, and stretch goals? We will determine the success of our model by the accuracy of the translation and recognition of words, as well as the efficiency of the model. We are planning to run our model on a smaller dataset and then apply it to a larger dataset once we have fine tuned our model. We are going to compare our novel idea about using fewer layers and more augmentation to increase efficiency to existing models to judge its efficiency and accuracy. Base Goal: Creating a working model that is more efficient than existing models used in ASL recognition Target Goal: Achieving similar accuracies to the MS-ASL paper Stretch Goal: Live interpretation of ASL

Ethics:

What broader societal issues are relevant to your chosen problem space?

A broader societal issue that is relevant to our chosen problem is the lack of representation for ASL, including the lack of translation for ASL in real time or for videos. We have chosen this problem to be able to identify an area for change through a deep learning model that is able to read what ASL is being performed in a video and being able to use what the model learns in other situations.

What is your dataset? Are there any concerns about how it was collected, or labeled? Is it representative? What kind of underlying historical or societal biases might it contain?

Although the dataset seems ethically sourced, there are a few aspects to consider. Although the participants in the videos intended to make their work public, they did not necessarily expect to be used to train a network. Additionally, due to the lack of racial representation in the dataset, it is possible that the final product could perform worse on people of color, since it has less exposure to the relevant training examples.

Division of Labor: Briefly outline who will be responsible for which part(s) of the project.

Colden will be responsible for preprocessing the data and compiling various datasets of fewer classes to train the models on initially. Once this is complete, he will help with the model training and fine-tuning. Anushka and Kaitlyn will be responsible for developing the architecture for the Conv2+1D, Conv3D and Conv-LSTM models We are all going to work on the evaluation of models, finetuning, interpretation, and write-up of our results.

References

[1] Joze, H. and Koller, O. “MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language” Microsoft https://export.arxiv.org/pdf/1812.01053

[2] J. Carreira and A. Zisserman, "Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 4724-4733, doi: 10.1109/CVPR.2017.502.

[3] D. Maturana and S. Scherer, "VoxNet: A 3D Convolutional Neural Network for real-time object recognition," 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany, 2015, pp. 922-928, doi: 10.1109/IROS.2015.7353481.

[4] Tran, Du, et al. "A closer look at spatiotemporal convolutions for action recognition." Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018.

Check In 3

Introduction:

Developing robust tools for sign language classification can help alleviate the communication gap between the hard-of-hearing community and those unfamiliar with ASL. Deep learning is an ideal approach to this problem as it is able to capture complex underlying patterns in high-dimensional data such as videos. An algorithmic approach of classifying ASL clips is infeasible, due to the significant variation between signers and the high complexity of the data. Though currently, models such as CNN-LSTMs, 3-D CNNs, hidden markov models etc. aim to tackle this problem, there does not seem to be a consistent approach. In this project, we aim to compare three convolutional architectures, Conv2+1D, Conv3D and Conv2D-LSTM models in an efficient manner by reducing the number of trainable parameters involved. The Conv2D-LSTM approach is a potential novel architecture, different from a CNN-LSTM layer, which can be used for sign language classification which may offer improved accuracy and insights given the spatial and temporal nature of sign language videos.

Challenges: What has been the hardest part of the project you’ve encountered so far?

The hardest part of this project has been downloading the data because it is so large and takes a long time, and we have had to explore some memory efficient ways to process the data because the program kept timing out. To do so, we stored the numpy arrays into h5 file formats. We are also exploring using the GPU for our model computations. Additionally, it has been challenging to adapt 2D models to 3D videos, trying to adapt the convolutional networks to 3D videos.

Insights: Are there any concrete results you can show at this point? How is your model performing compared with expectations?

Given our models’ lightweight architectures we do not expect concrete performance results yet compared to the MS-ASL baseline models without significant architecture tweaking and increased memory and compute power. Instead, we aim to compare the three architectures' performance. Currently, the Conv2Plus1D, 3D CNN and ConvLSTM models are underfitting. We will need to update the architecture to improve performance results. We do expect that the ConvLSTM architecture will perform better than the other two given that it uses temporal reasoning which is important for video classification.

Plan: Are you on track with your project? What do you need to dedicate more time to? What are you thinking of changing, if anything?

We are currently training the three model architectures on a subset of our original training dataset. The subset consists of the top 5 classes in the dataset. After completion, time will be dedicated towards training the models on the entire training dataset with validation and testing. Depending on the results, model architectures will be expanded on. Final steps will involve condensing the model results into a set of figures representing the training, validation and testing accuracies. We will also include metrics on efficiency metrics (number of trainable parameters), given our aim is to develop efficient models for sign language video classification.